DEV Community 👩‍💻👨‍💻

Discussion on: How To Scrape Amazon at Scale With Python Scrapy, And Never Get Banned

iankerins profile image
Ian Kerins Author

To scrape those extra pricing details you will need to find the selectors for them and add those field to the item.

When I open that page, I don't see the fields as Amazon is probably only showing them based on the geography you are in.

So if you create new selectors for those prices you want and add them to the item, then you can update the mysql storage pipeline to store that data as well.