From Rails to Elixir: Safe Rewrites with NOOP Deployments

This is the second of a series of articles about my journey going from Ruby on Rails to Elixir/Phoenix.
\Part-1 From Rails to Elixir: Know Your App
\Part-2 From Rails to Elixir: Safe Rewrites with NOOP Deployments

Recap

I'm porting a Price Tracker App originally built with Ruby on Rails to Elixir/Phoenix. My main motivation is to reduce infrastructure costs as I'm currently paying 24 USD/month for a virtual machine on Google Cloud, the smallest possible to support the whole system. Please read Part-1 for a comprehensive write-up on why I decided to do this migration.

Refactor Anxiety

Being able to improve your existing software for the better is always a good thing and often times a rare privilege. Improvements may come in different forms: you can add missing tests for a less tested feature, modernize and tighten up security by upgrading dependencies, refactor a chunk of code for all the good reasons and sometimes even port an entire piece to a more suitable technology that will enhance flexibility/reliability/performance.

It's not all roses, though. Replacing something that is currently working and serving its purpose often comes with the great responsibility of having to deliver a surrogate that works at least as well as the legacy stuff. There's also the moment of the switch over, which require careful planning, partial shutdowns and in some cases it even demands for a full stop and restart of the entire system. Never a good thing.

These great concerns of 1) not wanting to disrupt the running system during the transition phase and 2) wanting the new replacement to deliver what is expected; are two quite dramatic sources of developer anxiety.

Stress Less with the 6 R's

During my previous experience maintaining a steadily growing product, I learned a bunch about continuously evolving a 5 years old codebase without affecting the ongoing business. As a small team, we couldn't afford unexpected failures in production provoked by reckless deployments that would steal everyone's attention and effectively halt development.

For that reason, we were always very thoughtful about how refactors or any change in general would go into production.

The informal techniques we used varied depending on the type and scale of the migration coming ahead. They would usually fall into one of the 6 Application Migration Strategies - The 6 R's formulated by Amazon AWS in the context of the Cloud:

1\ Rehosting
2\ Replatforming
3\ Repurchasing
4\ Refactoring (re-architecting)
5\ Retire
6\ Retain

The line that separates them is fuzzy but I find the list very useful to clearly define the boundaries of the work I have in hands. It gives me clarity of thought and helps me plan the initiative. Let's say we are moving our infrastructure away from Amazon AWS and into Google Cloud - that's rehosting and therefore we should take every precaution inherent to moving one system from one place to the other. It makes it sound simple, and that is reassuring.

I would also recommend employing one, and only one, migration strategy at a time. If you need to rehost from AWS into Google Cloud and also replatform from Ruby on Rails into Elixir/Phoenix, then do one after the other and don't try to do it all at the same time. It might be tempting to do both at once but if you really think about it, that's effectively deploying a predominantly unknown piece of software in an equally unknown environment. Even with proper testing an staging servers in place, results can be unpredictable.

In my experience, migration success is inversely proportional to the underlying complexity. If you can, make sure that you keep your migrations as simple and unitary as possible.

NOOP Deployments

Perhaps my favorite technique of them all is what I call (and maybe someone before me?) the NOOP Deployment.

NOOP (No Operation) is a programming concept used to describe an action that literally does nothing. Its origin goes back to the Assembly language where it acts as an instruction that has no effect in the system. Wikipedia knows better.

So what exactly is a NOOP Deployment? It's a deployment technique where the new code is placed where it should live in production, but in an inactive state. I'm going to stretch this idea even further by deploying a completely blank Elixir app to the production infrastructure, with no custom code whatsoever, and let it live in there in an inert state while implementation progresses.

To illustrate this idea, the following picture shows exactly how I feel when I have to spend weeks or even months working on an application from top to bottom and the time finally comes to ship it to production:

It doesn't seem right, does it? You don't know exactly if the new terrain will be capable of supporting the structure, or if the old materials will play well in the new environment. Now imagine that there are other houses next to it, you certainly don't want to bring them down along if a catastrophe happens while you're unloading your traveling house. It's not going to do any good to your neighborhood credit.

That's exactly how migrations can go wrong. Deploy a defective application into a production infrastructure close to components of a running system and all hell breaks loose. Think databases going bonkers with connection bursts, external services being bombarded with requests or any other unexpected behavior.

For the sake of my own sanity, I prefer to do it more like this:

Find the terrain, lay the foundations and build upon. A lot less can go wrong when development is mindful of infrastructure and good things will arise when implementation is done in sync with deployment.

Martin Fowler also builds on this idea by introducing the StranglerApplication. It's a concept inspired by the strangler vines in Australia, which "seed in the upper branches of a fig tree and gradually work their way down the tree until they root in the soil". This is a great metaphor for subsystems that are created on the edge of old systems and grow gradually, slowly taking over.

I'm going to follow this approach to port my Ruby on Rails app to Elixir/Phoenix. The plan is to generate a blank Elixir application and place it next to the running Rails app. They will share the same virtual machine as the main goal of the project is to keep a slimmed down version of the existing server.

The Current State of Elixir Deployments

I've heard José Valim in a podcast (around minute 10:00) talking about the recent future of the language and saying that the one thing he still wants to resolve is deployment. He believes that deployment is so important that it should be a concern of the language itself.

Pursuing this idea, he joined forces with Paul Schoenfelder from Dockyard, creator of Distillery, in order to unify ideas from the community, test them in Distillery and eventually integrate a polished version into the Elixir core. Distillery is a rewrite of Erlang's release handling for Elixir and seems to be, from the perspective of a true outsider, one of the most popular solutions for deploying Elixir apps.

This comprehensive article written by Paul explains how they are planning to test their ideas in Distillery 2.0 and moving them into Elixir core once ready. It is very clear that configuration is their main concern, they want to get that right before it reaches the core of the language.

I saw other alternatives available, but to be honest they didn't catch my attention as much as Distillery, even more so after learning that its most recent version is basically a precursor for what's going to be available in the standard lib.

Production Release with Distillery 2.0

At this point, I'm expecting something similar to Capistrano, the go-to deployment thing for Ruby on Rails people. Add it as a dependency, generate a few files, customize it as needed and run a command to put it en route to the destination.

Distillery seemed very complete at first glance. It has lovely docs that guide you through the installation process as well as a few guides to follow for different use cases. The one about umbrella projects came in handy.

So, drop it like it's hot:

defp deps do
  [
    {:distillery, "~> 2.0"}
  ]
end

Install and generate default configuration files:

$ mix release.init

This gave me a rel/config.exs file with a few pre-defined values that can be customized as desidered. The most important lines might be:

environment :prod do
  # ...
end

release :shopxir do
  # ...
end

In my case, it generated a release with the same name as the umbrella project (shopxir) and included the only child application so far - jobs. This child application is a blank/empty Elixir app. There's nothing in there yet.

It also defines two environments, dev and prod. The latter is configured to include the Erlang runtime in the production release by default and doesn't include the source code.

Finally, run the command to build the release:

$ MIX_ENV=prod mix release

Which will fill _build/prod/ with a bunch of files and folders. This command has a nice output showing a few possible further actions one can do with the newly compiled release such as starting in the background, starting in the foreground and even a way to connect interactively with the running app (remote_console).

Just to be sure that my release is well packed, I'm going to add a simple static module to the jobs child app that I can call after the deployment in order to verify if it went as expected.

defmodule Ping do
  def ping do
    IO.puts "Pong."
  end
end

Rebuild and reconnect with remote_console:

$ MIX_ENV=prod mix release
$ _build/prod/rel/shopxir/bin/shopxir start # in the background
$ _build/prod/rel/shopxir/bin/shopxir remote_console
...
iex(shopxir@127.0.0.1)1> Ping.ping
Pong.
:ok

We're rolling 👍

I'm not going to discuss finer details about how releases can be customized because I'm not yet very familiar with them. As implementation goes and new features are moved from Rails into Elixir, I'll certainly need to dive deeper into this setup, but I'm making the decision to do it incrementally and as needed. For now, all I want is a live Elixir app placed in the server.

Deployment

Now, one thing that Distillery doesn't do is the actual heavy lifting of transporting the new version into the production infrastructure, the termination of the running app and initialization of the new version. They offer a few guides based on stuff like Docker and systemd, which is ok-ish, but I was looking for a more off-the-shelf tool. This was when I discovered that I'm a spoiled Ruby kiddo 👶

Docker is cool, I use it a lot locally to run external services such as PostgreSQL and Redis during development. Together with docker-compose, it makes it easy to organize external dependencies per project. For production though, it just feels difficult: configuration management, build servers, makefiles, orchestration, registries, discovery, Dockerfile, swarm, Kubernetes, etcd... I'm tired already.

If Docker in production is your thing, please take it easy on me. I probably just need to learn it properly.

On the other hand, Capistrano and its plugins did everything for me in the Ruby world. This thing takes care of downloading the latest release from a git repository, keeps a collection of releases in separate folders, compiles assets, runs migrations, stops/(re)starts the application and background workers, notifies monitoring services and much much more in a very simple way with almost zero configuration. It's also highly extensible.

I wanted something similar in the Elixir world but I didn't find anything so complete and straightforward.

Luckily, there's a very similar tool called Ansistrano based on Ansible (the provisioning framework) that is heavily inspired in Capistrano as the name says. It fits my needs because I'm already using Ansible for server provisioning.

Ansible has been a constant in all my projects since a few years ago. I use it primarily for infrastructure provisioning but it can be used for deployments too. It's a very versatile tool, people use it to do many different things related to manipulation of remote or even local machines.

In summary, Ansistrano will do the following:

1\ Build a new release locally using Distillery
2\ Create a folder in the remote server and copy the new release into it
3\ Symlink the new folder so that it is interpreted as the current release
4\ Create/update a systemd service that will execute the application
5\ Flip the switch to the new release by restarting the corresponding systemd service

Common deployment tasks such as running database migrations and asset compilation are expected to be handled using Distillery hooks, that's why they are not present in the previous list.

systemd is running the app in foreground mode and sends a SIGTERM by default for termination. This is enough for now because there are no running tasks such as HTTP requests being handled by Phoenix or other long-running processes in the background.

The whole thing (including template files) has a little over 50 lines of Ansible declarations. Check them out in this Gist.

Summary & Next Up

System rewrites are hard and a great source of developer anxiety. They require careful planning and a prudent approach.

Reduce anxiety and mitigate risks by streamlining the migration. Employ techniques such as the 6 R's to create focus and the StranglerApplication to incrementally replace the legacy system without harm. Use NOOP deployments to feel safe and reassured about your first deployment. Build upon.

Distillery is great for packaging Elixir releases but other tools like Ansistrano and systemd must be there to help with the actual deployment on the remote server.

Considering that I'm brand new to this, my first experience deploying an Elixir application was reasonably pleasant. I still feel a few knowledge gaps and I'm also pretty sure that my current setup is too simple for what's coming, but I'm happy with starting with a basic approach and growing it as needed.

Now that I have deployed my Elixir app in production, it's time to think about which parts of the legacy Rails app should be migrated first. This is going to be an important decision that I will leave for the next article, but my brain is already thinking Background Jobs.

I hope that you have enjoyed the 2nd article of the series. I tried to be a little more technical by dropping a few code snippets and diving deeper into tools. I'm wondering if you take this as a positive thing.

As always, feedback is much appreciated 🙏🏻

Thanks for reading ❤️