The Adventures of Blink #15: Continuous Delivery? Deployment? Both???

Greetings! This week we're talking about the "other half" of CI/CD... which might be the other 2/3 of CI/CD, depending on how you want to slice it! I say this because CD is ambiguous within the DevOps community... it can mean Continuous Delivery or it can mean Continuous Deployment.

At first glance you might think, "Ben, you're splitting hairs here. Delivery and Deployment are the same thing, and you're making it complicated." And for a good bit of history in InfoTech, you'd be right!

Traditionally, when we were Waterfalling our code into Production, the cycle looked like this:

You'd have an idea, and you'd build a thing to implement that idea, and then you'd Deploy it. Deploy in this sense meant "deliver it to customers". And it made sense - your standalone app might come on a CD (if you're old) or a floppy disk (if you're oldER, like me) and it was physically "delivered". And when the internet became a thing, and we didn't have to use physical media like this anymore, we kept the same mindset because we didn't change our methodology. Code that was "deployed" to production was thereby "delivered" to customers because they could use it as soon as the deployment was finished.

But this came with PROBLEMS. Software teams in the waterfall would write code for months or years, go through an arduous deployment event, and then immediately start to get feedback from customers in the wild about things that were ultimately bad design decisions. It was MUCH too late to fix these issues properly because larger-scale problems would necessitate a complete redesign, and those kinds of problems were common when we delivered large amounts of change at once! So we started to experiment with Agile techniques... ways to deliver smaller units of value faster.

This caused a new problem: some of our customers were used to getting big, sweeping changes on the order of months/years apart, managed by a team of "release managers" who ensured that (a) everything was tested / ready for production, and (b) upcoming changes were communicated to customer groups who might be affected by changes in program flow. When we started delivering updates weekly, or even daily, these updates became overwhelming - more quantity of change alerts led to notification fatigue and more changes meant retraining customers more frequently (and perhaps expensively!).

We couldn't just "slow down" the progress. The whole point of Agile was to reduce how much time it took to get work done! Making software teams wait for the release schedule meant tying up software engineers as they had to shepherd changes through the process, sometimes weeks or months after it was written. In doing so they became less effective because they lost the context of that work while having more and more things pile up behind.

The solution here is to separate the meanings of two words (as implied in the title and opening): Delivery is the act of releasing the software to customers while Deployment is the act of changing the code in production.

Uh, Ben, if you change production you're releasing the code, aren't you?

Traditionally, yes. my changed code is being put in production, so I'm releasing it. It's time to pick up a couple of techniques that will help us to further separate the meanings of our two words:

Feature Flags.

A Feature Flag is a programming technique where you wrap a changed bit of logic in an IF statement that's tied to an environment variable, like this:

if RELEASE_ID_12345 {

   // do the new behavior

} else {

   // do the old behavior

}

This has some great benefits for the long-term behavior of your app for several reasons:

• "Delivering" your new code to customers is just changing a configuration bit from False to True. That means you don't have to actually redeploy the application and risk downtime.
• If a bug escaped into production, you can revert to the old behavior by flipping the bit back, and fixing the problem without affecting your customers.
• Your operations teams will be thrilled at the ability to control the application without having to run deployment processes - those are disruptive and risky compared to (change bit to FALSE) and they'll be able to prevent active severe outage tickets caused by code changes... or even other reasons. If you're dependent on a third-party service for something, and that service goes down outside of your control, wouldn't it be awesome to "fail gracefully" and provide 90% service to your customers while it's being fixed, rather than being completely down?

There's one other technique you have to nail down for feature flags to be this powerful and useful, though:

0-downtime deployments.

You have to be able to push code into production in ways that prevent the end-user from knowing that code is being deployed right now! How would you do this?

• Blue-Green deployment patterns. Blue-green is when you have two clusters of hardware for your production system... an "active" and a "standby". You use a load balancer to control whether "Blue" or "Green" is the "active" cluster (the other one, is of course, automatically the "standby") and then you do the deployment to the "standby" cluster. This allows you to validate that the deployment succeeded and the application is up and running before you swap the two clusters' positions - "standby" becomes "active" and vice-versa.

• Rolling deploys. In modern cloud architecture, we don't like having multiple clusters online all the time because it's expensive - so instead we do a "rolling-deploy" strategy in which a "deployment" creates new pods/containers/etc with the new code, and adds them to the cluster - and then as workloads complete on the old pods, they're killed off.

Continuous Deployment and Delivery

The meaning of Continuous Deployment is simple: When code is completed, it is put into a continuous integration pipeline that tests it, and if the tests pass it's deployed to production. Simple, straightforward... but only a slice of the whole proverbial pie.

Continuous Delivery, on the other hand... is a little more involved, a bit too much to cover here. If you're super interested and want to read some of the research/thought leadership on this, check out MinimumCD.org. The folks there have done a tremendous job of mapping out what "minimum viable continuous delivery" should mean for an organization.

Wrapping up

CD is a multifaceted acronym, even within the DevOps community. It's important that we learn the specific terms we're using and explore the nuances that make them different from one another. I hope you've come to an understanding of Continuous Delivery and Continuous Deployment... tune in next week as we explore another "Continuous" concept: Continuous Testing!