Don't Couple Your Deployments

You’ve made changes to your service and go to deploy it. Among your changes, you added a new SQS queue.

Your deployment pipeline applies the IaC changes to get you that new queue. Then, it updates your code.

Later, there’s a problem with your code, so you roll back. However, rolling back the commit will also revert your SQS queue and cause your IaC to destroy it, which you don’t want.

This is a contrived example, but I’ve seen variations of it so many times.

Another common example is Kubernetes deployments using kustomize. Typically, a directory with manifests in a repository is applied using kustomize in the pipeline.

Developers add configuration in the form of environment variables to the Deployment. A new service version is deployed and needs to be rolled back.

You want to revert the commit, but the environment variables must stay as the new version so you can’t. You can’t simply revert everything in all cases. For example, say the application’s database host changed because of an outside migration, and the old host has been torn down.

Just like with application code, don’t couple your deployments too tightly. Treat configuration as a separate deployment from application code.

Join the 80/20 DevOps Newsletter

If you're an engineering leader or developer, you should subscribe to my 80/20 DevOps Newsletter. Give me 1 minute of your day, and I'll teach you essential DevOps skills. I cover topics like Kubernetes, AWS, Infrastructure as Code, and more.