DEV Community

Nicolae Rotaru
Nicolae Rotaru

Posted on • Originally published at page2api.com

The easiest way to scrape eBay products

Introduction

This article will describe the easiest way to scrape eBay products with Page2API


eBay is an online shopping site that's best known for its auctions and consumer-to-consumer sales.

You can scrape eBay if you intend to perform tasks such as:

  • analyzing products
  • predicting market trends
  • price monitoring

Compared to Amazon, scraping eBay can be a little more challenging, but the product page doesn't require a real browser, which leads to faster scraping speed.

For scraping eBay, we will use Page2API - a powerful and delightful API that makes web scraping easy and fun.

We will scrape 2 types of eBay pages:

  • Products list page
  • Product page

Prerequisites


To perform this simple task, you will need the following things:

The free trial offers a credit that covers up to 1000 web pages to scrape.

  • A product or a category of products that we are about to scrape.

In our case, we will search for 'fitbit 4 silicone strap'
and then scrape the product page for a random product.

Scraping eBay Products List Page


First what we need is to type 'fitbit 4 silicone strap' into the search input from eBay's search page change the view type to Gallery View.

This will change the browser URL to something similar to:

https://www.ebay.com/sch/i.html?_nkw=fitbit+4+silicone+strap&_sacat=0&_dmd=2&rt=nc
Enter fullscreen mode Exit fullscreen mode

The URL is the first parameter we need to perform the scraping.

The page that you see must look like the following one:

eBay results page


If you inspect the page HTML, you will find out that a single result is wrapped into a div that looks like the following:

eBay result component

The HTML for a single result element will look like this:

eBay result HTML

The last part is the pagination handling.

In our case, we must click on the Next (→) button while the list item's class will be active:

Next page available

And stop our scraping request when the Next (→) button became disabled.

In our case, a new attribute (aria-disabled="true") is assigned to the Next (→) button:

Next page disabled

Now it's time to prepare the request that will scrape all products that the search page returned.

Setting the api_key as an environment variable

export API_KEY=YOUR_PAGE2API_KEY
Enter fullscreen mode Exit fullscreen mode

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.ebay.com/sch/i.html?_nkw=fitbit+4+silicone+strap&_sacat=0&_dmd=2&rt=nc",
      "real_browser": true,
      "merge_loops": true,
      "scenario": [
        {
          "loop" : [
            { "wait_for": ".pagination__next" },
            { "execute": "parse" },
            { "execute_js": "document.querySelector(\"a.pagination__next\").click()" }
          ],
          "stop_condition": "document.querySelector(\"a.pagination__next\").ariaDisabled == \"true\""
        }
      ],
      "parse": {
        "items": [
          {
            "_parent":"ul.srp-grid li.s-item",
            "title":"h3.s-item__title >> text",
            "link":"a.s-item__link >> href",
            "price":".s-item__price >> text",
            "shipping":".s-item__shipping >> text"
          }
        ]
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
Enter fullscreen mode Exit fullscreen mode

The result

    {
      "result": {
        "items": [
          {
            "title": "For Fitbit Charge 3/4 Replacement Silicone Wristband Straps Sports Watch Band",
            "link": "https://www.ebay.com/itm/174485572199?hash=item28a0268a67:g:1QMAAOSwLA9fjTwQ",
            "price": "$3.89",
            "shipping": "Free shipping"
          },
          {
            "title": "Unisex Silicone Sports Bracelet Strap Fashion Band for Fitbit Inspire/Inspire HR",
            "link": "https://www.ebay.com/itm/324593039446?hash=item4b9340b856:g:5jsAAOSwRLJgh62j",
            "price": "$3.49",
            "shipping": "Free shipping"
          },
          ...
        ]
      },
      ...
    }
Enter fullscreen mode Exit fullscreen mode

Scraping eBay Product Page


From the products list page, we need to click on any product.

The browser URL will look like this:

 https://www.ebay.com/itm/114503614713
Enter fullscreen mode Exit fullscreen mode

The URL is the first parameter we need to perform the product data scraping.

Let's prepare the request that will scrape the needed information from the page.

Setting the api_key as an environment variable (if needed)

export API_KEY=YOUR_PAGE2API_KEY
Enter fullscreen mode Exit fullscreen mode

Running the scraping request with cURL

    curl -v -XPOST -H "Content-type: application/json" -d '{
      "api_key": "'"$API_KEY"'",
      "url": "https://www.ebay.com/itm/114503614713",
      "parse": {
        "name": "h1 >> text",
        "price": "span[itemprop=price] >> text",
        "condition": "div[itemprop=itemCondition] >> text",
        "categories": [
          "#vi-VR-brumb-lnkLst span >> text"
        ],
        "amount_sold": ".soldwithfeedback a >> text",
        "seller_name": "#RightSummaryPanel span.mbg-nw >> text",
        "item_location": "span[itemprop=availableAtOrFrom] >> text",
        "feedback_score": "#RightSummaryPanel span.mbg-l a >> text",
        "approximate_price": "#convbidPrice >> text"
      }
    }' 'https://www.page2api.com/api/v1/scrape' | python -mjson.tool
Enter fullscreen mode Exit fullscreen mode

The result

    {
      "result": {
        "name": "Details about Fitbit Charge 2 3 4 Bracelet Stainless Steel Spare Band Nylon Milanese Sport",
        "price": "GBP 5.66",
        "condition": "New with tags",
        "categories": [
          "Jewelry & Watches",
          "Watches, Parts & Accessories",
          "Watch Accessories",
          "Wristwatch Bands"
        ],
        "amount_sold": "",
        "seller_name": "fro-shop",
        "item_location": "Mülheim, Germany",
        "feedback_score": "28166",
        "approximate_price": "US $7.76(including shipping)"
      },
      ...
    }

Enter fullscreen mode Exit fullscreen mode

Conclusion


That's it!

Scraping eBay product data is a bit tricky because of the HTML structure, but that is not a problem if you have the proper scraping tool, such as Page2API that makes web scraping something you can enjoy.

The original article can be found here:

https://www.page2api.com/blog/scrape-ebay-products/

Discussion (2)

Collapse
rajasekharguptha profile image
Rajasekhar Guptha

Nice one @nrotaru
Stress more on coding explaantion (try explaining snippets of code rather than hard coding it and explaining the process )

Collapse
nrotaru profile image
Nicolae Rotaru Author

Make sense!

I'll try to describe the process in more detail next time :)