DEV Community

Karoline Alencar ✨
Karoline Alencar ✨

Posted on • Originally published at karolinealencar.com

Tutorial: The Sims 4 Price Scraping with NodeJS, Puppeteer & Nodemailer

Hi everyone! :)

In today's post I will teach you guys how to make a The Sims 4 Price Scraping. That means we are going to enter the EA website and extract the prices from their HTML and send to our email a list with the pack title, original, discount and final price.

Scraping is really cool to play with and you can do fun things with that! You can create your own scraping following this post logic.

So let's begin!

Installing NPM packages

We need Puppeteer and Nodemailer in your project, so let's install!

    npm install nodemailer puppeteer
Enter fullscreen mode Exit fullscreen mode

Let's create an index.js file and require puppeteer and nodemailer

const puppeteer = require("puppeteer")
const nodemailer = require("nodemailer")
Enter fullscreen mode Exit fullscreen mode

Let's begin with the fun part!

Create an async function called scrape

const scrape = async () => {}
Enter fullscreen mode Exit fullscreen mode

Tell puppeteer to open EA page

const scrape = async () => {
  const browser = await puppeteer.launch({ headless: false })
  const page = await browser.newPage()
  await page.goto("https://www.ea.com/games/the-sims/the-sims-4/pc/store")
  await page.waitForSelector(".product__group__pack")
}
Enter fullscreen mode Exit fullscreen mode

Here we create an browser instance, open a page and go to EA website. Then we wait for it to open and load.

Extract price and pack title

    const scrape = async () => {
        ...

        const result = await page.evaluate(() => {
            const products = Array.from(document.querySelectorAll('.product__group__pack'));

            const productsInfo = products.map(product => {
                const original = product.querySelector('.pack--hover__price--original');
                const discount = product.querySelector('.pack--hover__price--discount span');
                const final = product.querySelector('.pack--hover__price--final span');

                return {
                    title: product.querySelector('h2').innerText,
                    price: {
                        original: original ? original.innerText : null,
                        discount: discount ? discount.innerText : null,
                        final: final ? final.innerText : null,
                    }
                }
            });

            return productsInfo;
        });

        browser.close();
        return result;
    }
Enter fullscreen mode Exit fullscreen mode

Here we get all packs title, original, discount and final price. Note that we need to look at the HTML structure to see how we can get that specific data. In that case they have a specific class for the pack.

We then return a object with the title and price from each pack. After that we close the browser and return the array.

Now that we have all packs info, let's create our function to send an email!

Create the email send function

const emailSend = async products => {}
Enter fullscreen mode Exit fullscreen mode

This function will receive the product array that we created in the last section.

Add email and password info

const emailSend = async products => {
  const email = "fake@gmail.com"
  const password = "fakepass"
}
Enter fullscreen mode Exit fullscreen mode

Here you can add your email and password. Note that it will only work with gmail and you need to enable 'allow less secure apps' in your config, don't forget to turn it off later.

Create nodemailer transporter

    const emailSend = async products => {
        ...
        const transporter = nodemailer.createTransport({
          service: 'gmail',
          auth: {
              user: email,
              pass: password
          }
      });
    }
Enter fullscreen mode Exit fullscreen mode

Here we create a nodemailer transport with the user and password.

Create email content

    const emailSend = async products => {
        ...
        const mailOptions = {
            from: email,
            to: email,
            subject: 'The Sims 4 Prices',
            html: `
                <h1>The Sims 4 Prices</h1>
                <ul>
                    ${products.map(product =>
                        `<li>
                            <h2>${product.title}</h2>
                            <ul>
                                ${
                                    product.price.original
                                    ?
                                    `<li>
                                        <p>
                                            <b>Original:</b> ${product.price.original}
                                        </p>
                                    </li>`
                                    : ''
                                }
                                ${
                                    product.price.discount
                                    ?
                                    `<li>
                                        <p>
                                            <b>Discount:</b> ${product.price.discount}%
                                        </p>
                                    </li>`
                                    : ''
                                }
                                ${
                                    product.price.final
                                    ?
                                    `<li>
                                        <p>
                                            <b>Final:</b> ${product.price.final}
                                        </p>
                                    </li>`
                                    : ''
                                }
                            </ul>
                        </li>`
                    ).join('')}
                </ul>
            `
        };
    }
Enter fullscreen mode Exit fullscreen mode

Here we create the email HTML. It contains all packs that we got in the scrape function with its title and price info.

Send email

    const emailSend = async products => {
        ...
        await transporter.sendMail(mailOptions, function (error, info) {
            if (error) return console.log(error)
            return console.log(info);
        });
    }
Enter fullscreen mode Exit fullscreen mode

Here we tell nodemailer to send the email.

Now lets execute the code.

Executing scrape and emailSend function

scrape().then(data => emailSend(data))
Enter fullscreen mode Exit fullscreen mode

Here we execute the scrape function and pass its result to emailSend. And that's all! =) Now we need to execute that file in nodejs.

Executing the index.js file in nodejs

Go to the index.js folder and execute that command in the terminal

    node index.js
Enter fullscreen mode Exit fullscreen mode

Now check your email! 😛

Final code

const puppeteer = require("puppeteer")
const nodemailer = require("nodemailer")

const scrape = async () => {
  const browser = await puppeteer.launch({ headless: false })
  const page = await browser.newPage()
  await page.goto("https://www.ea.com/games/the-sims/the-sims-4/pc/store")
  await page.waitForSelector(".product__group__pack")

  const result = await page.evaluate(() => {
    const products = Array.from(
      document.querySelectorAll(".product__group__pack")
    )

    const productsInfo = products.map(product => {
      const original = product.querySelector(".pack--hover__price--original")
      const discount = product.querySelector(
        ".pack--hover__price--discount span"
      )
      const final = product.querySelector(".pack--hover__price--final span")

      return {
        title: product.querySelector("h2").innerText,
        price: {
          original: original ? original.innerText : null,
          discount: discount ? discount.innerText : null,
          final: final ? final.innerText : null,
        },
      }
    })

    return productsInfo
  })

  browser.close()
  return result
}

const emailSend = async products => {
  const email = "fake@gmail.com"
  const password = "fakepass"

  const transporter = nodemailer.createTransport({
    service: "gmail",
    auth: {
      user: email,
      pass: password,
    },
  })

  const mailOptions = {
    from: email,
    to: email,
    subject: "The Sims 4 Prices",
    html: `
                <h1>The Sims 4 Prices</h1> 
                <ul>
                    ${products
                    .map(
                      product =>
                        `<li>
                            <h2>${product.title}</h2>
                            <ul>
                                ${
                                product.price.original
                                  ? `<li>
                                        <p>
                                            <b>Original:</b> ${product.price.original}
                                        </p>
                                    </li>`
                                  : ""
                              }           
                                ${
                                product.price.discount
                                  ? `<li>
                                        <p>
                                            <b>Discount:</b> ${product.price.discount}%
                                        </p>
                                    </li>`
                                  : ""
                              } 
                                ${
                                product.price.final
                                  ? `<li>
                                        <p>
                                            <b>Final:</b> ${product.price.final}
                                        </p>
                                    </li>`
                                  : ""
                              } 
                            </ul>
                        </li>`
                    )
                    .join("")}
                </ul>
            `,
  }

  await transporter.sendMail(mailOptions, function(error, info) {
    if (error) return console.log(error)
    return console.log(info)
  })
}

scrape().then(data => emailSend(data))
Enter fullscreen mode Exit fullscreen mode

Top comments (0)