Project address: https://github.com/ray-d-song/web-archive
Why build this tool
I have been a loyal user of ArchiveBox for a long time. ArchiveBox is a very good web archiving tool, but it requires self-hosting and has high server requirements (requires headless browser). I used a Raspberry Pi before, and the performance was not good.
And for websites like x and Medium, which require login, ArchiveBox needs to manually configure tokens or cookies, which is troublesome.
So I thought, can there be a web archiving tool that doesn't require self-hosting, doesn't require headless browser, has no requirements for server, and can be cross-platform? Then I can access my archived pages anywhere, anytime, on any device.
Why Cloudflare
Cloudflare's Workers service is very powerful and free, with plenty of D1 databases and R2 storage buckets, which is very suitable for building this tool.
More importantly, Cloudflare's ecosystem is complete, supports one-click deployment and data migration. Cloudflare's global CDN service can also be used.
What can this tool do
- [x] Folder classification
- [x] Page preview image
- [x] Title keyword search
- [x] Showcase, share the pages you captured
- [x] Mobile support
- [x] Tag classification system
- [x] Read mode
How it works
web-archive is composed of the following parts:
- Browser extension: Save the page as a webpage snapshot and upload it to the server.
- Server: Receive the snapshot and metadata uploaded by the browser extension, and store them in the database and storage bucket.
- Web client: Query the snapshot and display it.
I used SingleFile's open-source code to save the page as a single html file (even including images and videos).
The server is completely based on Cloudflare's Workers service, with D1 database for storing metadata and R2 storage bucket for storing snapshots.
Although the number of interfaces is not small, I did not use ORM, actually I tried prisma and drizzle, because they caused a lot of trouble for deployment, so they were not used in the end.
The web client is built with React, Vite, TailwindCSS, and shadcn/ui, and the packaged size is astonishingly small, only 1.5MB. The packaged product will be embedded in the assets folder of the server, so it does not need to be deployed separately when deploying the server.
Limitations
I really like Cloudflare's free services, but there are some limitations.
- The CPU calculation time of a single request cannot exceed 10 milliseconds, otherwise it will be forcibly terminated. (I was surprised to find that the paid account is 30 seconds 😂)
- The memory usage cannot exceed 256MB, otherwise it will be forcibly terminated.
These limitations have affected the construction of the website to some extent, such as ssr or dom parsing during crawling.
However, no matter how it is said, thank you, Cloudflare!
Top comments (0)