DEV Community 👩‍💻👨‍💻

Cover image for How to scrape Google Maps Reviews?
Darshan
Darshan

Posted on • Updated on • Originally published at serpdog.io

How to scrape Google Maps Reviews?

In this post, we will learn to scrape Google Maps Reviews.

Requirements:

Before we begin, we have to install everything we may need in this tutorial to move forward.

  1. Node JS
  2. Unirest JS
  3. Cheerio JS

So before starting, we have to ensure that we have set up our Node JS project and installed both the packages - Unirest JS and Cheerio JS. You can install both packages from the above link.

We will use Unirest JS for extracting our raw HTML data and Cheerio JS for parsing our extracted HTML data.

Target:

Eiffel Tower Google Maps Results

Eiffel Tower Google Maps Reviews
We will target to scrape the user reviews on Eiffel Tower.

Process:

Now, we have set up all the things to be needed for preparing our scraper. We will use an npm library Unirest JS to make a get request to our target URL, so we can get our raw HTML data. Then we will use Cheerio JS for parsing the extracted raw HTML data.

We will target this type of URL:

`https://www.google.com/async/reviewDialog?hl=en_us&async=feature_id:${data_ID},next_page_token:${next_page_token},sort_by:qualityScore,start_index:,associated_topic:,_fmt:pc`
Enter fullscreen mode Exit fullscreen mode

Where,
data_ID - Data ID is used to uniquely identify a place in Google Maps.
next_page_token - The next_page_token is used to get the next page results.
sort_by - It is used for sorting and filtering results.

The various values of sort_by are:

  1. qualityScore - the most relevant reviews.
  2. newestFirst - the most recent reviews.
  3. ratingHigh - the highest rating reviews.
  4. ratingLow - the lowest rating reviews.

Now, the question arises how do we get the Data ID of any place?

Eiffel Tower Google Maps URL

https://www.google.com/maps/place/Eiffel+Tower/@48.8583701,2.2922926,17z/data=!4m7!3m6!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0!8m2!3d48.8583701!4d2.2944813!9m1!1b1
Enter fullscreen mode Exit fullscreen mode

You can see, in the URL the part after our !4m7!3m6!1s and before !8m2! is our Data ID.
So, our data ID in this case is 0x47e66e2964e34e2d:0x8ddca9ee380ef7e0
Our target URL should look like this:

https://www.google.com/async/reviewDialog?hl=en_us&async=feature_id:0x47e66e2964e34e2d:0x8ddca9ee380ef7e0,next_page_token:,sort_by:qualityScore,start_index:,associated_topic:,_fmt:pc
Enter fullscreen mode Exit fullscreen mode

Copy this URL in your browser and press enter. You will see, that a text file will be downloaded, by entering this URL in your browser. Open this file in your respective code editor. Convert it into a .html file. After opening the HTML file, we will search for the HTML tags of the elements we want in our response.

We will first parse the location information of the place, which contains - location name, address, average rating, and total reviews.

Eiffel Tower Google Maps HTML

From the above image, the tag for our location name is .P5Bobd, the tag for our address .T6pBCe, tag for our average rating is span.Aq14fc and tag for our total number of reviews is span.z5jxId.

All done for the location information part, we will now move towards parsing Data ID and next_page_token.

Search for the tag .lcorif. In the above image you can find the .lcorif tag in the second line. Under this tag, we have our tag for Data ID as .loris and of next_page_token as .gws-localreviews__general-reviews-block.

Now, we will search for the tags which contain data about the user and his review.
Search for the tag .gws-localreviews__google-review.

Eiffel Tower User Review HTML

This tag contains all information about the user and his reviews.
We will parse the extracted HTML for the user's name, link, thumbnail, number of reviews, rating, review, and the images posted by the user.
This makes our whole code look like this:

const unirest = require("unirest");
const cheerio = require("cheerio");

const getReviewsData = () => {
  return unirest
    .get("https://www.google.com/async/reviewDialog?hl=en_us&async=feature_id:0x47e66e2964e34e2d:0x8ddca9ee380ef7e0,next_page_token:,sort_by:qualityScore,start_index:,associated_topic:,_fmt:pc")
    .headers({
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36",
    })
    .then((response) => {
      console.log(response.body)
      let $ = cheerio.load(response.body);

      let user = [] , location_info,data_id,token;

      $(".lcorif").each((i, el) => {
        data_id = $(".loris").attr("data-fid");
        token = $(".gws-localreviews__general-reviews-block").attr(
          "data-next-page-token"
        );
        location_info = {
          title: $(".P5Bobd").text(),
          address: $(".T6pBCe").text(),
          avgRating: $("span.Aq14fc").text(),
          totalReviews: $("span.z5jxId").text(),
        };
      });

      $(".gws-localreviews__google-review").each((i, el) => {
        user.push({
        name:$(el).find(".TSUbDb").text(),

        link:$(el).find(".TSUbDb a").attr("href"),

        thumbnail: $(el).find(".lDY1rd").attr("src"),

        numOfreviews:$(el).find(".Msppse").text(),

        rating:$(el).find(".EBe2gf").attr("aria-label"),

        review:$(el).find(".Jtu6Td").text(),

        images:$(el)
          .find(".EDblX .JrO5Xe")
          .toArray()
          .map($)
          .map(d => d.attr("style").substring(21 , d.attr("style").lastIndexOf(")")))
        })
    });
    console.log("LOCATION INFO: ")
    console.log(location_info)
    console.log("DATA ID:")
    console.log(data_id)
    console.log("TOKEN:");
    console.log(token)
    console.log("USER:")
    console.log(user)
    });
};

getReviewsData();
Enter fullscreen mode Exit fullscreen mode

Result:

Eiffel Tower Google Maps Scraped Data

Our result should look like this 👆🏻.
These are the results of the first ten reviews. If you want to get another 10 put the token, which we have found in our code in the below URL:

https://www.google.com/async/reviewDialog?hl=en_us&async=feature_id:0x47e66e2964e34e2d:0x8ddca9ee380ef7e0,next_page_token:tokenFromResponse,sort_by:qualityScore,start_index:,associated_topic:,_fmt:pc
Enter fullscreen mode Exit fullscreen mode

In this case, we have our token as CAESBkVnSUlDZw== .
You can find the reviews for every next page using the token from their previous pages.

With Google Maps Reviews API:

Serpdog | Google Search API offers you 100 free requests on sign-up.
Scraping can take a lot of time sometimes, but the already made structured JSON data can save you a lot of time.

const axios = require('axios');

axios.get('https://api.serpdog.com/reviews?api_key=APIKEY&data_id=0x89c25090129c363d:0x40c6a5770d25022b')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
    console.log(error);
  });
Enter fullscreen mode Exit fullscreen mode

Result:

Statue Of Liberty Scraped Reviews Data

Conclusion:

In this tutorial, we learned how to scrape Google Maps Reviews. Feel free to ask me anything in comments. Follow me on Twitter Thanks for reading!

Additional Resources:

  1. How to scrape Google Organic Search Results using Node JS?
  2. Scrape Google News Results?
  3. Scrape Google Images Results
  4. Scrape Google Autocomplete Suggestions Results

Author:

My name is Darshan and I am the founder of serpdog.io.

Top comments (2)

Collapse
anasnmu profile image
Anas Nabil

Great article, Thanks for sharing, I'd like to see more Advanced Scrapping Examples

Collapse
darshan0_1 profile image
Darshan Author

Thanks for reading the article sir!
Yeah, surely I will write more blogs on scraping Google.

🌚 Friends don't let friends browse without dark mode.

Sorry, it's true.