DEV Community

Discussion on: How To Scrape Amazon at Scale With Python Scrapy, And Never Get Banned

Collapse
 
smaug profile image
smaug

firstly thank you for answer.

'tutorial.pipelines.TutorialPipeline': 300 <<< I guess a comma is needed here.

secondly, the price information cannot be scraped in the above codes. what could be the reason for this?

For example, can we register mysql by scraping 3 different prices of the following product?

amazon.com/dp/B07KSJLQCD

List Price: $25.00
Price: $17.45
Lightning deal " if any"

MS -> My Telegram t.me/smesut

Thread Thread
 
iankerins profile image
Ian Kerins

To scrape those extra pricing details you will need to find the selectors for them and add those field to the item.

When I open that page, I don't see the fields as Amazon is probably only showing them based on the geography you are in.

So if you create new selectors for those prices you want and add them to the item, then you can update the mysql storage pipeline to store that data as well.