DEV Community 👩‍💻👨‍💻

DEV Community 👩‍💻👨‍💻 is a community of 963,864 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Cover image for Launching Crawlee, the web scraping and browser automation library for Node.js
Ondra Urban
Ondra Urban

Posted on

Launching Crawlee, the web scraping and browser automation library for Node.js

Hello world,

Today, drawing on our team's years of experience, we're launching Crawlee, the the web scraping and browser automation library for Node.js that's built for fastest development and maximum reliability in production.

Main features

🖼 Supports headless browsers with Playwright or Puppeteer

⚡️ Supports raw HTTP crawling with Cheerio or JSDOM

🎛 Automated parallelization and scaling of crawlers for top performance

🐾 Avoids blocking using smart sessions, proxies, and browser fingerprints

🚎 Simple management and persistence of queues of URLs to crawl

🗜 Written completely in TypeScript for type safety and code autocompletion

📚 Comprehensive documentation, code examples, and tutorials

💪🏼 Actively maintained and developed by Apify—we use it ourselves!

Getting started

Visit or run the following command:

npx crawlee create my-crawler
Enter fullscreen mode Exit fullscreen mode

Liked Crawlee?

💛 You can support the project on GitHub, Product Hunt, or Hacker News

Top comments (3)

joelbonetr profile image

Amazing! 😁

Do you plan on writing more posts using it to showcase?

I would love to see the typical stand-alone example of a google maps crawler using Playwright and Crawlee! 😍

mnmkng profile image
Ondra Urban Author

If you don't mind an actual production grade example then this Google Maps scraper uses Apify SDK v2, from which we spun Crawlee off. Apify SDK and Crawlee interface is almost the same.

Crawlee is essentially Apify SDK v3, without the things that could only be used on the Apify platform. If you wish, you can read here, why we did the spin off.

We will migrate the Google Maps scraper to Crawlee soon, to leverage some of the new features.

joelbonetr profile image

Than you for all this information, much appreciated 😁

This post blew up on DEV in 2020:

js visualized

🚀⚙️ JavaScript Visualized: the JavaScript Engine

As JavaScript devs, we usually don't have to deal with compilers ourselves. However, it's definitely good to know the basics of the JavaScript engine and see how it handles our human-friendly JS code, and turns it into something machines understand! 🥳

Happy coding!