From Cloud to Garage: My journey with games.directory

Before we dive in, just a heads up – this isn't a story of me saving millions or buying £500k worth of fancy servers.
It's a simple, modest, story of how I moved my pet project, games.directory, from the cloud to my garage, running it on 3 Mac Minis. Plus, I will also a give a tiny deep dive into some of the internals of games.directory and decisions that I had to make along the way.

--

I'm a big fan of 37signals and DHH. When they announced they're leaving the cloud, I was all ears.
Funny ( not really ) enough, I also lost my job around the same time and had to go into savings mode, literally and figuratively.

--

I had a problem. How do I keep games.directory online, with no job, with no income coming from it, and with a monthly AWS bill of ~£3500?

I laid out all of my AWS resources, from most expensive to least expensive, and tried to figure alternatives.

S3 (~£1000/m): Hosting images and RDS backups.

This is an important resource as it powers Splash! and all images are being served from it, with no CDN. ( I know, I know )

At the time, the bucket size was around 50TB, which is why it was the biggest spender with around £750 a month alone mainly on egress. Nowadays it's around 250TB.

OpenSearch (~£750/m): Powering ElasticSearch needs for efficient data exploration.

I used OS to power my ElasticSearch needs and offer flexible search across the board, without worrying about scaling my RDS instance.

ElasticCache (~£600/m): Used by ElasticSearch and Sidekiq for background processing.

Redis was used by both ElasticSearch for re-indexing, and Sidekiq for background processing.
I had to beef up on nodes to handle the load.

games.directory runs around 1 million jobs a day, ranging from indexing, image processing, and syncing with the various APIs we use.

RDS (~£500/m): Hosting MySQL needs.
EC2 (~£200/m): Hosting the main Rails app.

Apart from that, I also used VPC, Route53, IAM, WorkMail, and a few other minor services that didn't cost much.

--

Initially I looked at alternatives, with the goal of halving the bill at least.
While I know my shit around Linux, provisioning, and metal, I don't like it. DevOps is boring and I'm lazy. Plus, I knew moving off completely would be quite the undertaking.

Render: No MySQL and no alternatives as far as I can tell, at the time.
Fly.io: I really liked this.. felt very much like Heroku, and a lot cheaper. At the time they weren't offering everything I needed, so I had to pass.
Heroku: I love Heroku, and I still use it for other side-projects, but it would have been too expensive to run games.directory on it.
DigitalOcean: Too expensive.

I even tried GCP and Azure. The UI and complexity is just stupid. I spent a whole day on Azure and all I managed to do was launch a Windows machine.
They can both fuck the right off with their shit UI. Maybe I was too used to AWS. Cost wise, wouldn't have been any better.

So here I am, with no viable alternatives and with a beefy bill coming soon. As I mentioned, I also lost my job around the time and I was on the lookout for a new work station, for my freelance, as my main machine was provided by work.
M2 just released and I saw that a Mac mini was about £600 and based on its specs, I thought it could handle games.directory easily. So I bought one.

Next few days I setup the mac mini and got it ready by running the main rails app, connected to a local Redis server, exposed to the internet via Cloudflare.

Obviously, it worked! I had games.directory running from my mac mini, in my garage. At this time I was still connected to RDS, OpenSearch and S3 via AWS. They were 3 big problems that I still had to figure out.

OpenSearch

As I mentioned, OS allowed me to offer powerful searching and filtering features, but it was also used, primarily, to give Users a way to view their data without putting a strain on the database which was already under heavy load due to the API sync.

I'll use the Community page as an example; If I go to this page, I will get a list of all users in the system, that can be a few millions, but also, some of their stats. Like how many games or achievements they have overall.
The problem with this is mainly that the additional stats are not stored on the same table. So the Community page is a combination of a few tables:

• Users
• Networks ( polymorphic ) - [XBOX PlayStation Steam Epic Stadia]
• ActiveStorage ( Images )

In the ElasticSearch database, each User had their own document, with all this information combined, stored as a single JSON object. This allowed me to offer a fast and efficient way to showcase this data, without having to worry about joins, or N+1 queries.

I'm a big fan of ElasticSearch, but once your data grows, it can become difficult to maintain. As I continue to work on games.directory, I find new and better ways to sync data and provide Users with a richer experience.
What I really mean is, I change my mind too fucking much. So when the data I want to display changes, I have to reindex my clusters. This can be difficult when you have 100+ million rows.
This would take days and in that time, everything else had to pretty much take a backseat.

I know there are ways to fix this, batching, parallelisation etc.. but I'm a single developer, working on a pet project, and honestly, I just want to write code. I can't express how much I dislike DevOps.
At this point I figured that I couldn't fit an ElasticSearch server on the mac mini, nor did I really want to.

ElasticSearch has been a guilty pleasure for years, and it allowed me to offer these kind of features easily without having to worry about MySQL performance. But in all honesty it made me feel dirty, because it allowed me to be lazy and design inefficient databases.
I was using ES as a crutch, and I knew it.

With that in mind I decided to remove ES from the equation and go back to MySQL. 1st attempt went really fucking bad.

Moved everything to MySQL and the loading time went from less than a millisecond to HTTP 500 across the board, due to timing out.

Over the next few months I refactored a lot of the internals, code wise. There wasn't much I could do regarding database design, but I made good use of indexes, and counter caching. It's a trade-off though because this has increased the need for me to implement various hooks around the code to update these caches and counters. But it's a trade-off that I'm happy with.

So, 3 months later, and a lot of code changes, I was ready to remove ES. Not going to lie, those months were fucking hard, and I felt really out of my depth.
But I learnt a ton of new information regarding MySQL, Indexing, Caching, and most importantly, ActiveRecord.
I've been using Rails for 10+ years and I'm ashamed to say that I knew so little about the difference between includes and joins. Especially regarding performance.

I could go on for days regarding the stuff I learnt in those 3 months, and the refactoring that I had to undertake. Maybe another day..

As of now, ElasticSearch is no longer part of games.directory. I do miss its features, and I feel like adding new features to the app, is much slower since I have to take into consideration how this data will be displayed, and queried.

MySQL

My biggest problem with MySQL was its size. At the time, if my memory serves me right, it was around 250GB. The Mac Mini's SSD storage, with the default specs, is 256GB.
If there's one thing I have a-plenty though, is storage. I have a Plex server running on my rack with 100TB of storage, for, uhm, family pictures.

I initially trialed having the database on the rack and expose it via Cloudflare, and see if that was viable. It wasn't.

The performance was terrible.

After a lot of researching I found out:

• The round-trip from the app, to Cloudflare, and then back to my server, was a big bottleneck.
• The fact that my SSDs were primarily Archive and NVMes drives. I built my rack for serving media, so I didn't really care about performance. I just wanted a lot of storage. I do have 2 performant SSDs that are used for transcoding, but they are 100% in usage most of the time.

So, on a whim, I decided to buy a new Mac Mini PRO with a 10GB ethernet port and a 4TB SSD which would only run MySQL. I knew that all of this would be solved by having another machine running locally, with the performance of M2.
I was right. Additionally, having a 10GB ethernet connected directly to the rails app mac mini was a big plus.

Having the database on a mac mini proved to be very efficient and I was back to having a fast connection to my database. Also, my MySQL got a performance boost running on M2. No surprise there.

One big downside to this is, again, storage. My database has only increased in size after adding indexes and caching. As of today, it's now around 1.2TB and it's literally growing by the day. One proud papa here.
As I hoard more data, each day, the 4TB of storage on my Mac mini will not be enough. But, that's a problem for future me, fuck that guy.

So with OpenSearch out of the question, and MySQL running locally, I was left with S3.

S3

Problem with S3 has always been the egress. Splash! holds over, as of now, ~150TB of storage and around 20 billion images, with the rest being taken by ~500 million videos. Storing them locally wasn't a big issue, I did have to buy some additional storage though and change my rack size to accommodate them.

The problem was serving them. I knew that I couldn't serve them locally, as I would have to expose my rack to the internet, and I didn't want to do that.

Without throwing money at it, there was no way I could continue providing Splash! as a service, for free. Nor could I continue hoarding all these images and videos. Honestly, most of it it's just junk, but I have issues.

So I went down the route of building a solution. I wrote this fancy schmancy algorithm that allows me to have the best of both worlds.
I can store all the images I want locally, and serve them via CDN using Cloudflare by using a speculative algorithm that allows me to sort of predict which images will be requested and when.

I know that sounds like a lot of bullshit, but hear me out! It's actually pretty simple.

When a user wants to see their directory, there are certain pages that are always viewed. For example, my Accomplishments page.

The 1st page will always have my latest 25 accomplishments. All those images are already cached, no magic there.
By default, each user's latest anything that has images is being cached, forever, or until the data changes.
Now here's where the cough algorithm cough kicks in. As you scroll through the page, all the way to the bottom, the app knows you want to, potentially, navigate to the next page and kicks in the caching mechanism that will pre-cache the images required for the next page page as it knows what needs caching.

By caching I mean uploading the images to Cloudflare which will be served up via CDN.

These images, that are cached by the algorithm have a TTL of 1 hour, so the bucket will stay slim.

This is a fairly simple example. But you get the gist.

This is very similar to how Speculative Loading works, but it's tailored to work for my use case.

There's still a few kinks to iron out, but it's served me, no pun intended, well so far. Needless to say, this wouldn't be possible if I didn't have a 1GB upload speed. Otherwise, by the time my app serves the next page and the time it takes to upload the images and then serve them, the user would have been shown placeholders.

Doing it this way, allows me to keep my Cloudflare bucket slim, and most importantly, it allows me to keep the egress costs down.

--

This is it, everything is running locally, more or less, and I have no more cloud dependency apart from Cloudflare.

Today, games.directory runs on 3 Mac Minis in my garage. It has it's own dedicated internet connection, with a fallback on 5G mobile.
It costs me roughly £300 a month.

£150: Dedicated Lila Connect 1GB up/down
£30: 120GB top-up for fallback 5G connection
£50/£100: Cloudflare Image
£25: Cloudflare

I've been running it out of my garage for more than a year now. In that time, I saved around £20.000+ in AWS costs.
Needless to say I'm happy with the decision and the costs of buying the 'metal' have been paid off.

--

Of course, there's ups and downs to this.

Ups:

• I can finally feed my family now.
• Control and Security. I know exactly where my data is, and what's going on.
• I finally have an excuse to pay a ridiculous amount of money for a 1GB up/down connection.
• No more restrictions on what I can store. I can hoard all the data I want.

Downs:

• DevOps and hardware maintenance. I hate it, but I have to do it.
• It's been a hell of a year optimising the shit out of my app and ensuring, mainly, the database is efficient, and my queries are fast.
• Storage, Storage, Storage.. images, backups, caching, all take storage. My rack is growing every quarter. I will either have to stop being such a hoarder or buy a bigger house, to accommodate my racks.
• Noise, oh the fucking noise. I'm glad it's in the garage.
• Having the app in my garage leaves me at the mercy of my electricity and internet provider. Sure, I'm not running some important piece of software that 1 minute downtime will kill someone, but it still sucks to see my uptime monitor scream at me.
• My electricity bill keeps going up as I add more storage to my rack..
• Backing up my 'S3' is non existent, and I have no fallback if any of my SSDs, holding images, will fail. If it fails, I lose its contents. Right now, I don't have the capacity or the resources to have a 1 to 1 backup.
• Remote access. There's none. If I'm on holiday and I need access to the server for something, anything, I'm unable to.

Right now I'm not ready to expose my rack or servers to the world, even via SSH. I'm still learning about networking and homelabs and I'm not comfortable doing it, yet.
I wouldn't want my naked pictures to be leaked on the-fapenning.

Obviously, more downs than ups, but no takesies backsies now. It's done.

On the same topic, downs, when my machine, which at the time was one, went down for software/hardware related issues, I experienced the biggest downtime ever. And if that wasn't enough, I wasn't home either, so couldn't do shit.

22 fucking hours I was devastated.

At the time, I had not even thought about setting up stuff like auto-login and putting the server in a background process which would would have started it automatically on boot.

You learn from experience I guess. I ended up buying a 2nd Mac Mini, dockerizing the app, and running it via Docker Desktop.

Scalling, am I right? :))

Now, if one of the machines fails, the other one will pick the slack while the other one reboots and starts the server and whatnot. It's actually beautiful.

Fingers crossed they don't fail at the same time..

--

This is it, this my story. I'm still learning, and trying to come up with cheap solutions to my problems. I'm kind of obsessed with performance right now, so if you have suggestions, I would love to hear them!

If you read all of it, thank you. I hope you enjoyed it and if you leave a comment, hit that notification bell and subscribe, that would be great :))
But no, seriously, if you leave a comment I'd really appreciate it as I'm keen on starting to blog more, and want to make sure I'm on the right path.

--

My take on this whole ordeal, 5 stars, would do it again.
I haven't saved millions of pounds in the process, but £2500 a month for me, is a lot of money. Especially in today's economy.

And while it's easy to push a button on Heroku and magically have your app running, it's also easy to forget that there's a lot of stuff going behind the scenes, and, as a full stack, you should know about it, at the very least.

I learnt SO MUCH in the process, DevOps, Networking, Optimizations, Database design, and most importantly, I learnt a lot about myself and what I'm capable of, when it comes to money lol.

--

I'm being asked a lot, so here's my answer to:

Q: Wait, are you doing this for something that doesn't bring you any money?
A: Yes, games.directory is a passion project. I've been working on it since 2015, with one goal.

Provide a free solution to gamers. No ads, no premium* features, no bullshit. And I will continue to do so, until the end of the internet.

* I do offer premium features, like using your earned points to buy stuff, but come on, that's not something I can provide for free. I'm not a charity.