DEV Community


Faster Than Requests with Multi-Thread URL-to-SQLite Web Scraper

juancarlospaco profile image Juan Carlos ・1 min read

Alt Text

Faster Than Requests 0.9.6 🐍

✔️ 5 Web Scrapers, Built-in, One-Liner.
✔️ Multi-Threading mass Downloader.
✔️ GitHub Actions build from scratch on every Push.
✔️ GitHub Actions publish to PYPI on every Release.
✔️ Basic get(), post(), download(), etc.
✔️ Multi-Part data for all POST functions.
✔️ Complete documentation with examples.
✔️ Works on Alpine Linux and Dockers.
✔️ CI, PyPI, Docker, Tests, Fixes, etc.
✔️ 0 Dependency, Python 3.9 ready, native bare-metal C performance.

Multi-Threading Web Scraper

Alt Text

  • threads = True Multi-Threading, threads = False no Threads.

URL-to-SQLite Web Scraper, give it 1 URL, gives back 1 SQLite database

Alt Text

  • SQLite database can be visualized with any SQLite WYSIWYG, like
  • HTTP Headers are stored as Pretty-Printed JSON.
  • Date and Time are stored as Unix Timestamps.

Images and Photos Web Scraper

Alt Text

  • Can get the images from the new HTML5 <picture> element.

Gimme tha powah

  • Star faster_than_requests on GitHub!, awesome stuff incoming...


Discussion (2)

Editor guide
turningpro profile image

Sounds amazing. What are the draw backs of that approach?

prostomarkeloff profile image

Nim is good! Thank you for this library, but i still wanna ask, will you implement moar in it? because it hasn't still been like awesome requests.