DEV Community

Nicolae Rotaru
Nicolae Rotaru

Posted on • Originally published at page2api.com

The Easiest Way to Scrape Amazon Products and Reviews

Introduction

Amazon is a vast Internet-based enterprise that sells all kinds of goods either directly or as the middleman between other retailers and Amazon's millions of customers.

You can scrape their data to run different types of tasks such as:

  • competitor analysis
  • improving your products and value proposition
  • identifying market trends and what influences them
  • price monitoring

Luckily, Amazon is a website that is pretty easy to scrape if you have the right tools.

For this purpose, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

In this case, we will scrape 2 types of amazon pages:

  • Search results page
  • Product reviews page

Prerequisites

To perform this simple task, you will need the following things:

The free trial offers a credit that covers up to 1000 web pages to scrape, and it takes under 10 seconds to create the account if you sign up with Google.

  • A product or a category of products that we are about to scrape.

In our case, we will search for 'luminox watches'
and then scrape the reviews for a random product.

Scraping all products from the search page

First what we need is to type 'luminox watches' into the search input from amazon's search page.

This will change the browser URL to something similar to:

https://www.amazon.com/s?k=luminox+watches
Enter fullscreen mode Exit fullscreen mode

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

Amazon search page

If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

Amazon search result

The HTML for a single result element will look like this:

Search result HTML breakdown

The last part is the pagination handling.

In our case, we must click on the Next → button while the list item's class will be active:

Next page enabled

And stop our scraping request when the Next → button became disabled.

In our case, a new class (a-disabled) is assigned to the list element where the button is located:

Next page disabled

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY
Enter fullscreen mode Exit fullscreen mode

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/s?k=luminox+watches",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "watches": [
          {
            "_parent": "[data-component-type='s-search-result']",
            "title": "h2 >> text",
            "link": ".a-link-normal >> href",
            "price": ".a-price-whole >> text",
            "stars": ".a-icon-alt >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
Enter fullscreen mode Exit fullscreen mode

The result

  {
    "result": {
      "watches": [
        {
          "title": "Men's Luminox Leatherback Sea Turtle 44mm Watch",
          "link": "https://www.amazon.com/Luminox-Leatherback-Turtle-Giant-Black/dp/B07CVFWXMR/ref=sr_1_2?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-2",
          "price": "$223.47",
          "stars": "4.5 out of 5 stars"
        },
        {
          "title": "The Original Navy Seal Mens Watch Black Display (XS.3001.F/Navy Seal Series): 200 Meter Water Resistant + Light Weight Case + Constant Night Visibility",
          "link": "https://www.amazon.com/Luminox-Wrist-Watch-Navy-Original/dp/B07NYXV77C/ref=sr_1_3?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-3",
          "price": "$254.12",
          "stars": "4.3 out of 5 stars"
        },
        {
          "title": "Leatherback SEA Turtle Giant - 0323",
          "link": "https://www.amazon.com/Luminox-Leatherback-SEA-Turtle-Giant/dp/B07PBC31N8/ref=sr_1_4?dchild=1&keywords=luminox+watches&qid=1634327863&sr=8-4",
          "price": "$179.00",
          "stars": "4.3 out of 5 stars"
        },
        ...
      ]
    },
    ...
  }
Enter fullscreen mode Exit fullscreen mode

Scraping product reviews

First what we need is to click on the See all reviews link from the Product page.

This will change the browser URL to something similar to:

https://www.amazon.com/product-reviews/B072FNJLBC
Enter fullscreen mode Exit fullscreen mode

The URL is the first parameter we need to perform the reviews scraping.

The HTML from a single review will look like this:

Amazon review HTML breakdown

Luckily, the pagination handling is similar to the one described above, so we will use the same flow.

Now it's time to prepare the request that will scrape all reviews.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY
Enter fullscreen mode Exit fullscreen mode

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.amazon.com/product-reviews/B072FNJLBC",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".a-pagination li.a-last" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\".a-pagination li.a-last a\").click()" }
          ],
          "stop_condition": "document.querySelector(\".a-last.a-disabled\") !== null"
        }
      ],
      "parse": {
        "reviews": [
          {
            "_parent": "[data-hook='review']",
            "title": ".review-title >> text",
            "author": ".a-profile-name >> text",
            "stars": ".review-rating >> text",
            "content": ".review-text >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
Enter fullscreen mode Exit fullscreen mode

The result

  {
    "result": {
      "reviews": [
          {
            "title": "Great watch & easy to read in low light conditions",
            "author": "Paul E. Papas",
            "stars": "5.0 out of 5 stars",
            "content": "I'm a 60+ year old equestrian and outdoorsman. I was looking for a watch that could take the shock of firearm discharge ..."
          },
          {
            "title": "Not Water Resistant, impossible to get amazon help",
            "author": "Benjamin H. Curry",
            "stars": "2.0 out of 5 stars",
            "content": "This watch has a 2 year warranty from date of purchase however after not even on full year my adult son went swimming with it ..."
          },
        ...
      ]
    },
    ...
  }
Enter fullscreen mode Exit fullscreen mode

Conclusion

That's pretty much of it!

As mentioned, Amazon is a website that is pretty easy to scrape if you have the right tools.
This is the place where Page2API shines, making web scraping super easy and fun.

The original article can be found here:

https://www.page2api.com/blog/scrape-amazon-product-data

Discussion (0)