Google Kubernetes Engine, CircleCI and Traefik for a full-fledged GitOps platform in the cloud - Part 2

This is the second part of "Google Kubernetes Engine, CircleCI and Traefik for a full-fledged GitOps platform in the cloud" ! Part 1 is accessible here and mandatory for the following.

3. A first "whoami" toy project

In this section, we will illustrate how to use Traefik to expose multiple instances/branches/versions of a same app on subdomains. Suppose we chose to dedicate the whoami subdomain (e.g., whoami.mywebsite.com) to this project. Our goal is to expose the master version of the app on a master subdomain (e.g., master.whoami.mywebsite.com) and the changes of a feature1 branch on the feature1 subdomain (e.g., feature1.whoami.mywebsite.com).

For this example, we will use the containous/whoami Docker image as a dummy web application, which only serves some information about the server and the received request for any request on port 80.

Create a namespace for the project

kubectl create namespace whoami

Deploy the master version

Fill the following placeholders in the whoami-master descriptor:

• DOMAIN: the domain you own (e.g., mywebsite.com)

Then apply it:

kubectl apply -f whoami-master.yaml

This will instantiate 3 instances of the whoami Pod, create a Service to load balance between them, and expose this Service through an IngressRoute. Notice the use of the traefik TLS certificate resolver created in step 2, the request for a *.whoami.[DOMAIN] wildcard certificate and the use of the master.whoami.[DOMAIN] Host rule to route traffic to this URL to our Service. In just a few seconds, the certificate is generated and your application server is exposed securely on the master.whoami of your domain (try it!).

Deploy the feature1 branch

Fill the following placeholders in the whoami-branch1 descriptor:

• DOMAIN: the domain you own (e.g., mywebsite.com)

Then apply it:

kubectl apply -f whoami-branch1.yaml

This will instantiate 3 new instances of the Pod, create a new Service to load balance between them, and expose this Service through an IngressRoute. Notice again the use of the traefik TLS certificate resolver created in step 2, the request for a *.whoami.[DOMAIN] wildcard certificate (which is already managed by Traefik and will be reused) and the use of the branch1.whoami.[DOMAIN] Host rule to route traffic to this URL to our Service. Your new app instance, distinct from the master environment is already live (try it!).

Use a template

Notice the very few differences between the whoami-master and the whoami-branch1 descriptors: all occurences of master have simply be replaced by branch1. In real world use cases, the Docker image will probably be different too.

Take a look at the whoami-template descriptor template file. Env-like placeholders (e.g., ${INSTANCE_NAME}) are used. We will use the envsubst tool to instantiate our template.

Fill the following placeholders in the whoami-template descriptor template:

• DOMAIN: the domain you own (e.g., mywebsite.com)

Do not touch the env-like placeholders.

Set the required environement variables, then instantiate the template and apply it:

export INSTANCE_NAME=branch2
export IMAGE=containous/whoami:v1.5.0
envsubst < whoami-template.yaml > whoami-branch2.yaml
kubectl apply -f whoami-branch2.yaml

Quick, your branch2 instance is already up!

The envsubst approach will get you quite far, but look into Kustomize or Helm if you need a more advanced configuration management tool.

4. A "myapp" project with CI/CD

In this section, we will illustrate how the manual deployment described in step 3 can be integrated in about any CI/CD pipeline. Suppose we chose to dedicate the myapp subdomain (e.g., myapp.mywebsite.com) to this project. Our goal is to expose any branch X on the X subdomain (e.g., branch feat1 exposed on feat1.myapp.mywebsite.com, feat2 exposed on feat2.myapp.mywebsite.com, etc.).

For this example, we will build our own Docker image in a CircleCI pipeline. For the sake of simplicity, this guide will use the same containous/whoami image, but feel free to build your own for real!

Create a namespace for the project

kubectl create namespace myapp

Create a "myapp" project on GitHub

In the sample_project folder, fill the following placeholders in the myapp-template descriptor:

• DOMAIN: the domain you own (e.g., mywebsite.com)

Then push the content of the sample_project folder at the root of your GitHub repository (i.e., the .circleci folder, the Dockerfile and the descriptor template should be visible at the root of your repository).

Notice in particular the .circleci/config.yml build configuration file for CircleCI (more about the syntax here).

Create a CircleCI service account on GCP

This can easily be done in the Cloud Console.

Create a new CircleCI account with role Storage Admin and Kubernetes Engine Admin. Create and download a (JSON) key for this account. This technical user will have write access to the Google Container Registry (GCR) of you project and permission to deploy to your cluster.

Build the project on CircleCI

In CircleCI, add your project in the build list in the Add Projects tab (use the Add manually button, since you have already committed a .circleci/config.yml file).

In the Settings tab, find your project in the Projects sub-tab and open its settings.

In Environment Variables, create the following variables. They are used in the CircleCI configuration and in the descriptor template.

• GCP_PROJECT: The id of your GCP project (visible in the Cloud Console)
• GCP_AUTH: The content of the JSON key file created in the previous step.
• GCP_REGISTRY: The GCR registry of your choice (eu.gcr.io/[GCP_PROJECT] for a storage in Europe)
• GCP_ZONE: The GCP Zone in which your cluster is hosted (visible in the Cloud Console, e.g., europe-west3-a)
• GCP_KUBE_CLUSTER: The name of your Kubernetes cluster

Relaunch the build: the master branch is deployed on the master subdomain (e.g., master.myapp.mywebsite.com).

Create branches on your GitHub repository and push them... and let the magic happen! From source code to production in less than 20 seconds!

5. Bonus track: delete stale instances in Kubernetes

Now you know how to deploy an instance of your app per GitHub branch in Kubernetes... But how to delete stale instances in Kubernetes (instances of the app created for branches that are since deleted in GitHub)?

Add a second job in your CircleCI configuration

As you can see, it involves quite a bit of bash wizardry...

jobs:
  build:
    [...]
  delete-stale-ci-stacks:
    docker:
      - image: google/cloud-sdk
    working_directory: /home/circleci/myapp
    steps:
      - checkout
      - run:
        name: Authenticate with GCP
        command: |
          echo ${GCP_AUTH} > /home/circleci/gcp-key.json
          gcloud auth activate-service-account --key-file /home/circleci/gcp-key.json
          gcloud --quiet config set project ${GCP_PROJECT}
          gcloud config set compute/zone ${GCP_ZONE}
          gcloud --quiet container clusters get-credentials ${GCP_KUBE_CLUSTER}
      - run:
        name: Delete stale instances
        command: |
          KUBE_FEATURE_BRANCHES=`kubectl get po -n myapp -l 'app.kubernetes.io/name=myapp' -o jsonpath="{.items[*].metadata.labels['app\.kubernetes\.io/instance']}"`
          GIT_BRANCHES=`git branch --list --remote | sed 's/  origin\///g'`
          for kube_branch in $KUBE_FEATURE_BRANCHES; do
            echo "Checking branch ${kube_branch}"
            git_branch_still_exists=false
            for git_branch in $GIT_BRANCHES; do
              if [ "${git_branch}" == "${kube_branch}" ]; then
                git_branch_still_exists=true
                echo "Git branch ${git_branch} still exists, instance ${kube_branch} will not be deleted."
              fi
            done
            if [ "${git_branch_still_exists}" = false ]; then
              echo "Git branch corresponding to instance ${kube_branch} has been deleted, deleting instance..."
              kubectl delete all,ingressroute -n myapp -l app.kubernetes.io/instance=$kube_branch
            fi
          done

Add the job in a new workflow triggered by a CRON

workflows:
  version: 2
  build-test-and-deploy:
    [...]
  delete-stale-deployments:
    triggers:
      - schedule:
        cron: "0 0,6,12,18 * * *" # Every six hours
        filters:
          branches:
            only:
              - master
    jobs:
          - delete-stale-ci-stacks

Conclusion

For a toy project, the cost of this infrastructure is about $100 a month. For real-world projects with large architectures we average $500 per project per month, and the pay-as-you-go pricing model of the services we use guarantees this cost to be quite linear. This might look like a lot, but in fact it is ridiculously low when you take into account the billable time not spent on maintaining this type of platform ourselves (or being slowed down by it!).

As of this writing, we have been using these tools and techniques for all our projects at Zenika Labs for more than two years. Feel free to use it, share it, suggest improvements or ask any question!