Best way to run Migrations in Production

We cover some principles when writing and running migrations in production here https://dev.to/woovi/zero-downtime-database-migrations-at-woovi-cc7.

However we didn't make explicit how to exactly you should be running them.

How you should not run migrations in production

Let's first cover how you should not run migrations in production first, and then we are providing our solution that is more robust and safe.

Do not run migrations when doing a deployment.

You should not run your migration when doing a deployment.
Running migrations when doing a deployment is not safe.
You can run your backend service in many instances (pods in k8s), so you can't be sure on each instance you should be running the migration.

Even if you figure it out and solve the concurrency problem when running the migration, the migration can fail.
If the migration fails, what should you do? should you rollback the deployment? and you should deploy again to try the migration again? it does not sounds robust.

If you have a big table/collection the migration can take a long time, maybe hours, should you wait for hours to decide if a deployment should be made or not?

Do not run a migration using a script

You can port forward your databases to your localhost and run a migration from your computer. This sounds a good idea, but you could have the wrong set of environment variables set and run the migration in the wrong environment.
It will also be slow as you have more network latency.

Do not run a migration in CI

Running migrations in a CI/CD environment will need you to expose your databases and production environment to the internet or the CI/CD environment.
Usually all developers have access to the CI/CD, so they can easily get access to your production environment causing some trouble.

The safest way to run a migration in production

If you should not run migrations in all the above ways, how the hell should you be running them anyway.

What we built at Woovi is a Migration service.
Our migration service is a docker image that contains all our migration files.
We update our migration docker image using our CI/CD both for staging and production environment.
We have a deployment on kubernetes that run this docker migration image service.

This is our migration deployment Kubernetes definition

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: woovi-stg
  name: woovi-stg-migration
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
  selector:
    matchLabels:
      app: woovi-stg-migration
  template:
    metadata:
      labels:
        app: woovi-stg-migration
    spec:
      containers:
        - name: woovi-stg-migration
          image: <>.dkr.ecr.us-east-1.amazonaws.com/woovi-migration:latest          
          command: ["/bin/sh", "-c", "--"]
          args: ["while true; do sleep 30; done;"]

It will run the woovi-migration without any command.

To run a specific migration we run the command like this:

kubectl exec --stdin --tty -it -n=woovi-stg woovi-stg-migration -- /bin/sh -c "nohup node migration.js ${migrationName} ${rerun} > /proc/1/fd/1"

This will run a specific command in the woovi-migration container.
nohup will make the command to hang up, not wait for completion
> /proc/1/fd/1 will forward the stdout of the command to kubernetes logs output, so we can see migration logs and output.

We also instrumented our migration with Elastic APM to have observability in our Kibana.

In Resume

This approach has many benefits.
You don't expose your database and production environment.
You always run the migration in the right environment.
You can see migration logs as they are running.
You have observability.
It is faster, because their is not network latency.
If it fails you can easily retry.

This is only possible because we decoupled our migration process from our deployment process.

Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Photo by Sébastien Goldberg on Unsplash