Introduction
Amazon is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and Amazon's millions of customers.
You can scrape their data to run different types of tasks such as:
- competitor analysis
- improving your products and value proposition
- identifying market trends and what influences them
- price monitoring
Luckily, Amazon is a website that is pretty easy to scrape if you have the right tools.
For this purpose, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.
In this case, we will scrape 2 types of amazon pages:
- Search results page
- Product reviews page
Prerequisites
To perform this simple task, you will need the following things:
The free trial offers a credit that covers up to 1000 web pages to scrape, and it takes under 10 seconds to create the account if you sign up with Google.
- A product or a category of products that we are about to scrape.
In our case, we will search for 'luminox watches'
and then scrape the reviews for a random product.
Scraping all products from the search page
First what we need is to type 'luminox watches' into the search input from amazon's search page.
This will change the browser URL to something similar to:
https://www.amazon.com/s?k=luminox+watches
The URL is the first parameter we need to perform the scraping.
The page that you see must look like the following one:
If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:
The HTML for a single result element will look like this:
The last part is the pagination handling.
In our case, we must click on the Next → button while the list item's class will be active:
And stop our scraping request when the Next → button became disabled.
In our case, a new class (a-disabled) is assigned to the list element where the button is located:
Now it's time to prepare the request that will scrape all products that the search page returned.
Setting the api_key as an environment variable
export API_KEY=YOUR_PAGE2API_KEY
Running the scraping request with cURL
curl -v -XPOST -H "Content-type: application/json" -d '{
"api_key": "'"$API_KEY"'",
"url": "https://www.amazon.com/s?k=luminox+watches",
"real_browser": true,
"merge_loops": true,
"scenario": [
{
"loop" : [
{ "wait_for": ".a-pagination li.a-last" },
{ "execute": "parse" },
{ "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
],
"stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
}
],
"parse": {
"watches": [
{
"_parent": "[data-component-type='s-search-result']",
"title": "h2 >> text",
"link": ".a-link-normal >> href",
"price": ".a-price-whole >> text",
"stars": ".a-icon-alt >> text"
}
]
}
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
The result
{
"result": {
"watches": [
{
"title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
"link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
"price": "$223.47",
"stars": "4.5 out of 5 stars"
},
{
"title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
"link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
"price": "$254.12",
"stars": "4.3 out of 5 stars"
},
{
"title": "Leatherback SEA Turtle Giant - 0323",
"link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
"price": "$179.00",
"stars": "4.3 out of 5 stars"
},
...
]
},
...
}
Scraping product reviews
First what we need is to click on the See all reviews link from the Product page.
This will change the browser URL to something similar to:
https://www.amazon.com/product-reviews/B072FNJLBC
The URL is the first parameter we need to perform the reviews scraping.
The HTML from a single review will look like this:
Luckily, the pagination handling is similar to the one described above, so we will use the same flow.
Now it's time to prepare the request that will scrape all reviews.
Setting the api_key as an environment variable (if needed)
export API_KEY=YOUR_PAGE2API_KEY
Running the scraping request with cURL
curl -v -XPOST -H "Content-type: application/json" -d '{
"api_key": "'"$API_KEY"'",
"url": "https://www.amazon.com/product-reviews/B072FNJLBC",
"real_browser": true,
"merge_loops": true,
"scenario": [
{
"loop" : [
{ "wait_for": ".a-pagination li.a-last" },
{ "execute": "parse" },
{ "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
],
"stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
}
],
"parse": {
"reviews": [
{
"_parent": "[data-hook='review']",
"title": ".review-title >> text",
"author": ".a-profile-name >> text",
"stars": ".review-rating >> text",
"content": ".review-text >> text"
}
]
}
}' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
The result
{
"result": {
"reviews": [
{
"title": "Great watch & easy to read in low light conditions",
"author": "Paul E. Papas",
"stars": "5.0 out of 5 stars",
"content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
},
{
"title": "Not Water Resistant, impossible to get amazon help",
"author": "Benjamin H. Curry",
"stars": "2.0 out of 5 stars",
"content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
},
...
]
},
...
}
Conclusion
That's pretty much of it!
As mentioned, Amazon is a website that is pretty easy to scrape if you have the right tools.
This is the place where Page2API shines, making web scraping super easy and fun.
The original article can be found here:
https://www.page2api.com/blog/scrape-amazon-product-data
Top comments (0)