Minimize failed deployments with Argo Rollouts and Smoke tests

Argo Rollouts is a progressive delivery controller created for Kubernetes. It allows you to deploy your application with minimal/zero downtime by adopting a gradual way of deploying instead of taking an “all at once” approach.

Argo Rollouts supercharges your Kubernetes cluster and in addition to the rolling updates you can now do

• Blue/green deployments
• Canary deployments
• A/B tests
• Automatic rollbacks
• Integrated Metric analysis

In this article, we will see a simple example that combines blue/green deployments with smoke tests that decide if the new application version will be promoted or not.

How Blue/Green deployments work

Before we look at the magic sauce behind Argo Rollouts, it is important to know the basics behind Blue/Green deployments (also known as Red/Black).

In the beginning, there is only one version active (single color). Once a new application is deployed a second instance of the application is created (two colors). The critical point here is that the new color does not receive any live traffic. All users are still on the old version.

Once developers are confident that the new version is ready to go, all live traffic is switched to the second color. The big advantage here is that if something goes wrong, developers can simply switch the traffic back to the previous version. It is always faster to switch a load balancer than redeploying again.

After some time, the previous version can be discarded completely. This is mostly because of wasted resources, but not strictly necessary. In several variations, the previous color can stay behind as a failsafe.

Previewing the next color with Argo Rollouts

Now that we have seen how blue/green deployments work in general, let’s see what Argo Rollouts does with Kubernetes.

Argo Rollout is a Kubernetes controller and has several options for Blue/Green deployments, but the most interesting one is the “preview” service that allows you to test the new color before making the traffic switch to live users. You can easily install the operator in your cluster by following the instructions.

You can find the example application at https://github.com/codefresh-contrib/argo-rollout-blue-green-sample-app. It is a simple Java application with two pods and no other dependencies.

To perform the initial deployment, clone the git repository locally and edit the file blue-green-manual-approval/rollout.yaml by replacing CF_SHORT_REVISION with any available tag from dockerhub (for example 8b39c04)

Save the file and then perform the first deployment with

kubectl create ns blue-green
kubectl apply -f ./blue-green-manual-approval -n blue-green

Here is the initial state using the kubeview graphical tool.

We can see that the application has two pods and we also have two services pointing at them. The service named “rollout-bluegreen-active” is the one for live traffic. The one named “rollout-bluegreen-preview” is internal and currently should not serve any traffic. It also points to the same color as the live one.

If you also want to monitor things from the command line be sure to install the Argo rollouts CLI and run

kubectl argo rollouts get rollout spring-sample-app-deployment --watch -n blue-green

The next step is to create a new version of the application. You can either change again the image tag on the manifest (and run kubectl apply again) or simply use the shortcut offered by Argo Rollouts

kubectl argo rollouts set image spring-sample-app-deployment spring-sample-app-container=kostiscodefresh/argo-rollouts-blue-green-sample-app:9c7732e -n blue-green

The image tag you use is arbitrary. It just needs to be different than the current one. As soon as Argo Rollouts detects the change it will do the following:

1. Create a brand new replica set with 2 pods for the new color
2. Redirect the “preview” service to them

Here is how it looks in Kubeview:

The CLI will also report the second replicaset:

This is one of the most critical points in a blue/green deployment. At this point all live users are still on the old version of the application. For them, it is business as usual and they are completely unaffected by what is happening behind the scenes.

It is now your responsibility to check the next version by utilizing the “preview” service that Argo Rollouts offers. Maybe you need to look at the health checks, check its metrics or even better run some smoke tests that verify the correctness of the new version.

You can easily find the IP of the preview service with

kubectl get svc  rollout-bluegreen-preview -n blue-green

The application has some dummy endpoints at /actuator/health and /actuator/metrics that you can visit with your browser. We will automate the process with Codefresh in the next section.

Once everything looks good we can promote the deployment with

kubectl argo rollouts promote spring-sample-app-deployment -n blue-green

This is the moment of truth. This command will instruct Argo Rollout to switch live traffic (i.e. all your users to the new version).

As you can see both services now point to the new version. The old version is still kept around for quick rollbacks. After a specified amount of time (30seconds is the default value), Argo Rollouts will completely destroy the old version.

We are now back to the original state with just two pods of the new application version.

A new deployment will repeat the whole process.

If for some reason your smoke tests fail, and you don’t want to promote the new version you can abort the whole process with

kubectl argo rollouts abort spring-sample-app-deployment -n blue-green

You can also go back to a previous version with

kubectl argo rollouts undo spring-sample-app-deployment -n blue-green

Argo rollouts has more options that allow you to choose the number of versions that stay behind, the amount of time each version is kept, which version to rollback to, and so on.

See the whole rollout spec for more details.

Notice also that for simplicity reasons, the two services in the example are powered by Load Balancers. Check the pricing of your Cloud provider before running this example on your own and be sure to clean up everything after you finish.

Automating everything with a Codefresh pipeline

Now that we have seen how you can perform manually blue/green deployments, it is time to automate the deployment process with Codefresh pipelines.

The application comes with some simple integration tests (that require the application to run). These will be used as smoke tests.

Following best practices, the target of the smoke test is configurable (and not hardcoded to localhost or other name). This means that you can easily run the smoke tests with any target regardless of the actual IP of your Loadbalancer.

We will create the following pipeline:

This pipeline does the following:

1. Clones the source code of the application
2. Builds a docker image
3. Deploys the application by updating the Kubernetes manifests. Argo Rollouts sees the new manifest and creates a new “color” for the next version
4. Runs integration tests against the “preview” service created by Argo Rollouts. Live users are still on the previous/stable version of the application.
5. If smoke tests pass the new color is promoted and becomes the new active version
6. If smoke tests fail the new color is discarded the all live users are not affected in any way

Here is the pipeline definition:

codefresh.yml
version: "1.0"
stages:
  - prepare
  - build
  - deploy 
  - finish   
steps:
  clone:
    type: "git-clone"
    stage: prepare
    description: "Cloning main repository..."
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: "${{CF_BRANCH}}"
  build_app_image:
    title: Building Docker Image
    type: build
    stage: build
    image_name: kostiscodefresh/argo-rollouts-blue-green-sample-app
    working_directory: "${{clone}}" 
    tags:
    - "latest"
    - '${{CF_SHORT_REVISION}}'
    dockerfile: Dockerfile
  start_deployment:
    title: Deploying new color
    stage: deploy
    image: codefresh/cf-deploy-kubernetes:master
    working_directory: "${{clone}}"
    commands:
      - /cf-deploy-kubernetes ./blue-green-manual-approval/service.yaml 
      - /cf-deploy-kubernetes ./blue-green-manual-approval/service-preview.yaml   
      - /cf-deploy-kubernetes ./blue-green-manual-approval/rollout.yaml   
    environment:
      - KUBECONTEXT=mydemoAkscluster@BizSpark Plus
      - KUBERNETES_NAMESPACE=blue-green 
  run_integration_tests:
    title: Smoke tests
    stage: deploy
    image: maven:3.5.2-jdk-8-alpine
    working_directory: "${{clone}}" 
    fail_fast: false
    commands:
     - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository verify -Dserver.host=http://13.86.102.74  -Dserver.port=80
  promote_color:
    title: Switching traffic to new color
    stage: finish
    image: kostiscodefresh/kubectl-argo-rollouts:latest
    commands:
      - /app/kubectl-argo-rollouts-linux-amd64 promote spring-sample-app-deployment -n blue-green --context "mydemoAkscluster@BizSpark Plus"
    when:
      steps:
      - name: run_integration_tests
        on:
        - success 
  abort_deployment:
    title: Keeping the existing color
    stage: finish
    image: kostiscodefresh/kubectl-argo-rollouts:latest
    commands:
      - /app/kubectl-argo-rollouts-linux-amd64 argo rollouts undo spring-sample-app-deployment -n blue-green --context "mydemoAkscluster@BizSpark Plus" 
    when:
      steps:
      - name: run_integration_tests
        on:
        - failure

You now have true Continuous Deployment! All deployments are fully automatic and “bad” application versions (that don’t pass the smoke tests) will never reach production.

Argo Rollouts has many more capabilities that we will see in the next posts of this series. For more details on Blue/Green deployment as well as an example with manual approval see the documentation page.

New to Codefresh? Create Your Free Account today!