Design Report - Migrations in Elm Fullstack Deployments

This post was originally published at https://michaelrätzel.com/blog/design-report-migrations-in-elm-fullstack-deployments

Elm Fullstack is a tool for developing web services and full-stack web applications. It leverages the Elm programming language and its ecosystem of libraries. As it evolves, Elm Fullstack automates more and more activities in software development that humans had performed in the past.

Some of the tasks it automates are related to maintaining an application's program state. This 'state' contains all information a web service should remember. It is the 'database' of the application. Maintaining this program state is one of the features of Elm Fullstack that shields app developers from the complexity of lower-level concepts in software development.

One common machine-level concept is the distinction between volatile and persistent memory. Many frameworks for software development reflect this distinction, delegating responsibility to handle this complexity to the app developer. The first feature I implemented in Elm Fullstack resolves this distinction so that we don't need to consider it when programming apps.
After persistence, I took on Migrations - another aspect of state management. This post overviews exploration of this area and the design I discovered for migrations in deployments.

Migrations

What are migrations, and why do we need them?

Successful software applications evolve. This evolution manifests in changes to our code and our data. After working on and testing functionality in a development environment, we eventually make our changes available to users and customers. To do this, we update a production system with the new program code. This process is what we call a 'deployment'.

In many cases, we can deploy an Elm app simply by pushing the code to a production system. But sometimes, a change affects the state of our backend or how we represent it in the code. Changing the type of the persistent state presents a new challenge. Since we maintain the backend state across deployments, there is a type mismatch between our app's old and new versions. To resolve the type mismatch, we need to transform that data, map it from the old to the new type.
Elm Fullstack manages these changes through a mechanism called migration.

The example of a type mismatch illustrates why we need migrations. If considering only type-mismatches, I might have picked an API that expects migrations only when changing the state type.
But, looking closer at app development, I found more uses for migrations.

The meaning of the state value not only depends on type declarations but also on functions. Functions in the program code define how our service responds to clients and how these responses depend on the current state of the service. When we deploy a new program version, these dependencies of responses on the state can change, effectively changing the meaning of the service state, even if its type declaration stays the same.
Sometimes when testing during development, I change the app state or edit the database directly. The migration functionality would help here, packaging such an edit in an atomic transaction.
To support these scenarios, I went with a design that, by default, requires us to declare a migration function for every deployment explicitly.

Application Programming Interface

How do migrations work from the app developers' perspective?

Migrations happen as part of deployments, so we do not trigger them explicitly. By default, every deployment implies a migration. But sometimes, we prefer to throw away the current backend state instead of coding a migration. For these cases, the API offers an option to take the app state from an init function instead of looking for a migration. In the command-line interface, we see this option surface on the deploy command.

When deploying a new version of our app, we describe how to map values from the current to the new state type. Elm Fullstack then does the type-checking across these versions and rejects the deployment in case of a mismatch. If we provided a matching migration function with our deployment, Elm Fullstack performs the migration in an atomic transaction.

Ok, now we know the admin interface will yell at us if we don't supply a migration with the next deployment. But how do we declare the migration?

The Elm programming language already provides the elements to code a migration. A function with one argument is enough to describe the transformation. When applying the migration, the type of that argument needs to match the backend state type found in the currently running version. We need to maintain the declaration of that old type somewhere in the new program code so that standard Elm tooling continues to work. Compared to type-checking in Elm, the rules for custom types are more relaxed for migrations. As long as all tags are the same, we can substitute one custom type with another. In other words, type-checking in migrations does not consider the module names of custom types. This way, we can conveniently place all custom types we keep around for migrations under a common prefix.

We code the migration as an Elm function that takes the old state as input and returns the new state with the new backend state type. We place it into the deployed code in an Elm module with a particular name to declare the migration.

For optimal consistency and familiarity, the type of the migrate function is the same as in other update functions used on the backend, minus the event-specific first argument. We might see apps that should issue commands on migration, so the return type supports that, analogous to the update functions for application events. The difference to update functions for application events is that the state type on the input side can differ from the output side.

To summarize, here is the Elm syntax with the type annotation of a migration function:

migrate : OldBackendTypes.State -> ( Backend.Main.State, ElmFullstack.BackendCmds Backend.Main.State )

Conclusion

What is the state of migrations now?

Migrations are implemented and available with the latest releases.
To declare the migration function, we place it in the module Backend.MigrateState like this:

module Backend.MigrateState exposing (migrate)

import Backend.Main
import ElmFullstack


migrate : Backend.Main.State -> ( Backend.Main.State, ElmFullstack.BackendCmds Backend.Main.State )
migrate state =
    ( state, [] )

In this example, we see the trivial case where we don't need any transformation.

While introducing type-checked migrations was significant progress, we can still improve the developer experience around migrations.

An apparent future step is to add automation for coding the migration function. Extracting the backend type declarations from a previous version into a dedicated Elm module is the least we can do here.