DEV Community

loading...

Faster Than Requests with Multi-Thread URL-to-SQLite Web Scraper

juancarlospaco profile image Juan Carlos ・1 min read

Alt Text

Faster Than Requests 0.9.6 🐍

✔️ 5 Web Scrapers, Built-in, One-Liner.
✔️ Multi-Threading mass Downloader.
✔️ GitHub Actions build from scratch on every Push.
✔️ GitHub Actions publish to PYPI on every Release.
✔️ Basic get(), post(), download(), etc.
✔️ Multi-Part data for all POST functions.
✔️ Complete documentation with examples.
✔️ Works on Alpine Linux and Dockers.
✔️ CI, PyPI, Docker, Tests, Fixes, etc.
✔️ 0 Dependency, Python 3.9 ready, native bare-metal C performance.

Multi-Threading Web Scraper

Alt Text

  • threads = True Multi-Threading, threads = False no Threads.

URL-to-SQLite Web Scraper, give it 1 URL, gives back 1 SQLite database

Alt Text

  • SQLite database can be visualized with any SQLite WYSIWYG, like https://sqlitebrowser.org
  • HTTP Headers are stored as Pretty-Printed JSON.
  • Date and Time are stored as Unix Timestamps.

Images and Photos Web Scraper

Alt Text

  • Can get the images from the new HTML5 <picture> element.

Gimme tha powah

https://github.com/juancarlospaco/faster-than-requests#extras-go-beyond-requests

  • Star faster_than_requests on GitHub!, awesome stuff incoming...

👑

Discussion (2)

pic
Editor guide
Collapse
turningpro profile image
turningpro

Sounds amazing. What are the draw backs of that approach?

Collapse
prostomarkeloff profile image
prostomarkeloff

Nim is good! Thank you for this library, but i still wanna ask, will you implement moar in it? because it hasn't still been like awesome requests.