Decentralization is what makes so many of the Fediverse’s unique features possible. And it can also make running an instance totally unmanageable. But Fast Forward program member and Jortage maintainer, Una Thompson is using Fastly to help solve one issue that arises with decentralization: content duplication. We wanted to learn more about the issue and how Una is solving it, so we sat down with her to learn more.
The Fediverse is a wonderful place — it enables small but interconnected communities to moderate themselves and choose with whom they associate. It gives users the freedom to own their data and migrate between instances. But with the freedom that decentralized social networks provide, there are also tradeoffs.
Federation ends up flinging a lot of content, i.e. media objects, back and forth between servers. Whenever someone posts, their instance pushes the status and objects to the instances of everyone who follows the poster, and all those instances download it. Each instance then separately uploads to its storage provider, duplicating it by the number of the poster’s followers.
It’s a tough problem, and Una has solved it in a really neat way with Jortage. It's a communal project that provides object storage and hosting and deduplicates object storage for pool members, reducing everyone's overall storage costs. To ensure media stays fast despite the project being used on over 70 instances all over the world, Jortage utilizes Fastly to accelerate and cache downloads. With Fastly, Jortage has avoided investing in their own global servers, keeping costs even lower, and makes the experience of cruising #CatsOfMastodon faster and more engaging. :)
Hannah: What initially inspired you to create Jortage?
Una: It's one of those things that just kind of comes to you while doing something else; at some point, I wound up thinking about the way media federation works and had a "hmm, there's no way that'd actually work" idea, prototyped it, and it worked great. So I scaled it up and got some other instance admins in on it, and the rest is history.
At one point I got a DM from Gargron [Eugen Rochko, CEO of Mastodon] asking how on earth Jortage actually works, as FFmpeg and ImageMagick (used by Mastodon to process incoming media) are nondeterministic — honestly, I don't know. We shouldn't be getting results this good. :P
Hannah: So how does it work?
Una: The basic way the storage pool works is that every uploaded file is hashed, and a MariaDB database stores a mapping of hashes to paths; new files are then uploaded to our backend storage provider, Wasabi, using the hash as a filename. Requests to pool.jortage.com look up the path in the DB, and if a match is found, return a redirect to blob.jortage.com with the relevant hash. Pool points to a server I run with poolmgr, and blob just points directly to Wasabi.
The storage pool CDN connection was pretty simple. The most complex thing in my Fastly config is the song and dance involved in doing a redirect for the root path, and it's highly specific to the storage pool. I also had to enable Segmented Caching to make Jortage work with videos, as Mastodon’s upload limit is 40MB.
The software that powers Jortage is the custom poolmgr. It’s very much a “standing on the shoulders of giants” project. Without S3Proxy and Apache JClouds, it would've been a much much tougher project to pull off. So poolmgr exposes an S3-compatible API that Mastodon (as well as Pleroma, Akkoma, Misskey, etc) talks to. However, the vast majority of storage on a Mastodon instance goes to its S3 provider, so ~70 instances are feeling the benefit of Fastly regardless. (Some rough benchmarks I did while waiting for DNS propagation suggest that Fastly’s total download time is 2-5× faster than our previous vendor, including uncached requests.)
The heaviest parts of Mastodon can't really be solved with a CDN; things like the job manager (Sidekiq) and federation requests are actively made worse by a CDN — just look at all the instances running behind Cloudflare that regularly have mysterious federation issues. Media is all the big files, though.
Hannah: The heaviest bits of federation — do you think there is a solution to make federation requests and Sidekiq run more efficiently?
Una: I don't think there's any web-based service you can point to today and call "efficient" without a lot of asterisks. "RESTful" services that sling JSON back and forth are not efficient by any definition of the word. I'm certainly not an expert on ActivityPub, but in general, it’s not what you would call an efficient protocol.
Making a single post causes your instance to send that post to every instance that's following you. That, in turn, causes those instances to make a request back to your instance for the thread context, which then can result in more requests for posts. They'll also check for your current profile information, which will cause them to load the links in your profile looking for rel=me links to process verification. And, of course, if there's attachments, they all get downloaded. All of these create jobs in the Sidekiq queue, which in the default Mastodon config has a very low number of threads and runs in a single process. It's not even well-documented that multi-process Sidekiq is possible — a lot of people heard of for the first time via this blog post by Nora Tindall.
Of course that itself adds overhead (Sidekiq uses dizzying amounts of memory), so I've been interested in the TruffleRuby solution to keep it all in one process and provide a better GC.
If that back-and-forth request explosion could be replaced with eager data pushing in the initial federation it'd likely help. There's another Jortage project related to the link resolution issue, called jort.link.
Some of it has to do with the fact it runs on the Ruby interpreter, and one of my "whenever I get a chance" projects is to try running Mastodon under GraalVM via TruffleRuby. I feel this will be a major win for Sidekiq, as it introduces JIT compilation and true multithreading. A global interpreter lock makes parallelism very shaky in vanilla Ruby (also an issue with vanilla Python, and one of the main benefits of PyPy).
The centralization inherent to Jortage is a negative for sustainability but running truly independent services that scale costs a fortune.
Hannah: What are your tips for other devs who want to set up an instance?
Una: Think long and hard about it. Running an instance is setting up a community, and running a community is a long (essentially endless) commitment, and doing proper moderation is very hard. I run a closed instance, and you don't have to look far to see all the stories of people who went into running a public instance without being prepared for what that means.
If you want to learn more about Jortage or connect with Una, visit Jortage’s homepage and reach out on Fedi to say hi.
Top comments (0)