Building a Distributed Turn-Based Game System in Elixir

One of the best things about building web applications in Elixir is LiveView, the Phoenix Framework feature that makes it easy to create live and responsive web pages without all the layers people normally build.

Many great Phoenix LiveView examples exist. They often show the ease and power of LiveView but stop at multiple browsers talking to a single web server. I wanted to go further and create a fully clustered, globally distributed, privately networked, secure application. What's more, I wanted to have fun doing it.

So I set out to see if I could create a fully distributed, clustered, privately networked, global game server system. Spoiler Alert: I did.

What I didn't have to build

What I find remarkable is what I didn't need to build.

I didn't build a Javascript front end using something like React.js or Vue.js. That is the typical approach. Building a JS front-end means I need JS components, a front-end router, a way to model the state in the browser, a way to transfer player actions to the server and a way to receive state updates from the server.

On the server, I didn't build an API. Typically that would be REST or GraphQL with a JSON structure for transferring data to and from the front-end.

I didn't need other external systems like Amazon SQS, Kafka, or even just Redis to pass state between servers. This means the entire system requires less cross-technology knowledge or specialized skills to build and maintain it. I used Phoenix.PubSub which is built on technology already in Elixir's VM, called the BEAM. I used the Horde library to provide a distributed process registry for finding and interacting with GameServers.

As for Fly.io's WireGuard connected private network between geographically distant regions and data centers? I don't even know how I would have done that in AWS, which is why I've always given up on the idea.

What I did build

What I built was just a proof of concept, but I'm surprised at how it came together. I ended up with a platform that can host many different types of games, all of which:

• Can be multi-player
• Offer a Jackbox-style 4-letter game code system
• Have on-demand game and match creation
• With a fast, response UI

And, just one little extra detail: the platform supports multiple connected servers operating together in clusters. Elixir for the win!

I created this as an open source project on Github, so you can check it out yourself.

https://github.com/fly-apps/tictac

Technology

I've worked with enough companies and teams to imagine several different approaches to build a system like this. Those approaches would all require large multi-disciplinary teams like a front-end JS team, a backend team, a DevOps team, and more. In contrast, I set out to do this by myself, in my spare time, and with a whole lot of "life" happening too.

Here's what I chose to use:

• Elixir programming language – A dynamic, functional language for building scalable and maintainable applications.
• Phoenix Framework – Elixir's primary web framework
• Phoenix LiveView – Rich, real-time user experiences with server-rendered HTML delivered by websockets
• libcluster – Automatic cluster formation/healing for Elixir applications.
• Horde – Elixir library that provides a distributed and supervised process registry.
• Fly.io – Hosting platform that enables private networked connections and multi-region support.

Application Architecture

There are many guides to getting started with LiveView, I'm not focusing on that here. However, for context, this demonstrates the application architecture when running on a local machine.

The "ABCD" in the graphic is a running game identified by the 4-letter code "ABCD".

Let's walk it through.

1. A player uses a web browser to view the game board. The player clicks to make a move.
2. The browser click triggers an event in the player's LiveView. There is a bi-directional websocket connection from the browser to LiveView.
3. The LiveView process sends a message to the game server for the player's move.
4. The GameServer uses Phoenix.PubSub to publish the updated state of game ABCD.
5. The player's LiveView is subscribed to notifications for any updates to game ABCD. The LiveView receives the new game state. This automatically triggers LiveView to re-render the game immediately pushing the UI changes out to the player's browser.
6. All connected players see the new state of the board and game.

We need a game

I needed a simple game to play and model for this game system. I chose Tic-Tac-Toe. Why?

• It's simple to understand and play.
• Easy to model.
• Doesn't bog down the project with designing a game.
• Quick to play through and test it being "over".

I want to emphasize that this system can be used to build many turn-based, multi-user games! This simple Tic-Tac-Toe game covers all of the basics we will need. Besides, Tic-Tac-Toe was even made into a TV Show!

This is what the game looks like with 2 players.

The game system works great locally. Let's get it deployed!

Hosting on Fly.io

Following the Fly.io Getting Started Guide for Elixir, I created a Dockerfile to generate a release for my application. Check out the repo here:

https://github.com/fly-apps/tictac

The README file outlines both how to run it locally and deploy it globally on Fly.io.

What is special about hosting it on Fly.io? Fly makes it easy to deploy a server geographically closer to the users I want to reach. When a user goes to my website, they are directed to my nearest server. This means any responsive LiveView updates and interactions will be even faster and smoother because the regular TCP and websocket connections are just that much physically closer.

But for the game, I wanted there to be a single source of truth. That GameServer can only exist in one place. Supporting a private, networked, and fully clustered environment means my server in the EU can communicate with the GameServer that might be running in the US. But my EU players have a fast and responsive UI connection close to them. This provides a better user experience!

Here is what I find compelling about Fly.io for hosting Elixir applications:

• Secure HTTPS automatically using Let's Encrypt. I didn't do anything to set that up!
• Distributed nodes use private network connections through WireGuard.
• Nodes auto-clustered using libcluster and the DNSPoll strategy. (See here for details)
• Geographically distributed servers near my users are clustered together.
• This was the easiest multi-region yet still privately networked solution I've ever seen! (I have experience with AWS, DigitalOcean, and Heroku)

Conclusion

For a proof-of-concept, I couldn't be happier! In a short time, by myself, I created a working, clustered, distributed, multi-player, globe-spanning gaming system!

The pairing of Elixir + LiveView + Fly.io is excellent. Using Elixir and LiveView, I built a powerful, resilient, and distributed system in orders of magnitude shorter time and effort. Deploying it on Fly.io let let me easily do something I would never have even tried before, namely, deploying servers in regions around the globe while keeping the application privately networked and clustered together.

Whenever I've thought of creating a service with a global audience, I'd usually scapegoat the idea saying, "Well I don't know how I'd get the translations, so I'll just stick with the US. It's a huge market anyway." In short, I've never even considered a globally connected application because it would be "way too hard".

But here, with Elixir + LiveView + Fly.io, I did something by myself in my spare time that larger teams using more technologies struggle to deliver. I'm still mind blown by it!

What will you build?

Tic-Tac-Toe is a simple game and doesn't provide "hours of fun". I know you can think of a much cooler and more interesting multi-player, turn-based game that you could build on a system like this. What do you have in mind?