loading...
Cover image for 🍪 Screenshot or scrape webpages behind a login with JavaScript (in Node.js)

🍪 Screenshot or scrape webpages behind a login with JavaScript (in Node.js)

benjaminmock profile image Benjamin Mock ・2 min read

Super useful puppeteer snippets 💭💭💭 (8 Part Series)

1) 📷 How to take a screenshot of a webpage with JavaScript in Node.js (using puppeteer) 2) 📷 More on Screenshots with JavaScript in Node.js (emulating devices) 3 ... 6 3) 📷 More on Screenshots with JavaScript in Node.js (creating PDFs) 4) 🗄️ The easiest way to scrape a website with Javascript (in Node.js) 5) 🍪 Screenshot or scrape webpages behind a login with JavaScript (in Node.js) 6) 🧭 Programmatically taking Screenshots while navigating & searching on webpages using JavaScript (puppeteer) 7) 👟 Executing JavaScript in the page context of puppeteer 8) 🚀 Performance measuring of a webpage with JavaScript in Node.js using puppeteer

Let's imagine, that you want to programmatically screenshot your dev.to dashboard. We learned in part 1 how to do screenshots. But so far we only did screenshots of public pages. The dev.to dashboard is only available if you're logged in. No problem! That's actually an easy task because you can set cookies with puppeteer. The only thing you have to do is open the Dev Tools and copy the remember_user_token cookie. With the correct domain settings (I prepared these in the example) you can now access the webpage as if you were logged in. Actually your puppeteer session is logged in. This is possible for most of the pages because usually, a login relies on a cookie, which you can just copy. Try it yourself!

// npm i puppeteer
const puppeteer = require('puppeteer');

const cookie = {
  name: 'remember_user_token',
  value: 'YOUR_COOKIE_VALUE', // replace this!
  domain: 'dev.to',
  url: 'https://www.dev.to/',
  path: '/',
  httpOnly: true,
  secure: true,
};

// we're using async/await - so we need an async function, that we can run
const run = async () => {
  // open the browser and prepare a page
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // set the cookie with your user information to appear logged in
  await page.setCookie(cookie);

  await page.goto('https://dev.to/dashboard');
  await page.screenshot({
    path: 'my_dev_to_dashboard.png',
    fullPage: false, // set to true to get the whole page
  });

  // close the browser
  await browser.close();
};

// run the async function
run();

My dev.to Dashboard

Please consider following me, if you're interested in what else you can do with puppeteer and if you don't want to miss any of my upcoming articles in this series.

Thanks for reading!

Super useful puppeteer snippets 💭💭💭 (8 Part Series)

1) 📷 How to take a screenshot of a webpage with JavaScript in Node.js (using puppeteer) 2) 📷 More on Screenshots with JavaScript in Node.js (emulating devices) 3 ... 6 3) 📷 More on Screenshots with JavaScript in Node.js (creating PDFs) 4) 🗄️ The easiest way to scrape a website with Javascript (in Node.js) 5) 🍪 Screenshot or scrape webpages behind a login with JavaScript (in Node.js) 6) 🧭 Programmatically taking Screenshots while navigating & searching on webpages using JavaScript (puppeteer) 7) 👟 Executing JavaScript in the page context of puppeteer 8) 🚀 Performance measuring of a webpage with JavaScript in Node.js using puppeteer

Posted on Nov 25 '19 by:

benjaminmock profile

Benjamin Mock

@benjaminmock

I'm Ben, a Frontend Developer from Germany working at ebay

Discussion

markdown guide