Earlier this month I built and deployed my first ever GitOps Continuous Delivery pipeline using kubernetes and flux. Throughout the process I found myself wading through quite a bit of outdated and confusing documentation, so having made it to the other side I wanted to write up a walk-through of the steps I took in the hopes that other developers can have an easier time.

The Stack:

AWS EKS for the kubernetes cluster
HelmOperator by Flux for GitOps
Terraform + Cloudformation for infrastructure

The application I deployed to the cluster was a GraphQL API that lives in a docker image on AWS ECR, but this can be adapted to any docker container or other containerized build.

Since there are a lot of good resources on creating a kubernetes cluster specific to whatever service you are using (AWS, Google GKE, etc) I am going to start this walk-through with the assumption that you have an existing cluster that you want to deploy an image to.

Step 1: GitHub

You will need a dedicated GitHub repository for this process. This is where you will commit all of the following configuration (your helm charts, helm releases, etc), and it should be separate from whatever repository you are using for building the application that you are deploying.

Step 2: Setting up Fluxcd

Follow the instructions for installing Helm and Fluxctl

Then, use helm to install the fluxcd charts:

helm repo add fluxcd https://charts.fluxcd.io

Create a namespace on your cluster for flux-related pods to live:

I used 'fluxcd' but you can choose whatever namespace makes sense for you

kubectl create ns fluxcd

Create your flux pod (the GitHub repo + branch will be from Step 1):

helm upgrade -i flux fluxcd/flux --wait \
  --namespace fluxcd \
  --set git.url=git@github.com:<GITHUB REPO> \
  --set git.branch=<BRANCH>

If your repository is private you will also need to add the knownhosts flag:

--set-file ssh.known_hosts=$known_hosts_path

pointing to the path where your knownhosts are, usually in the ~/.ssh directory.

Step 3: The Helm Operator

The helm operator is a controller that works in tandem with flux and allows you to create HelmReleases that will deploy an image, as well as update the image to the latest version whenever Flux detects a change.

Apply the Helm Operator to your cluster:

kubectl apply -f https://raw.githubusercontent.com/fluxcd/helm-operator/master/deploy/flux-helm-release-crd.yaml

Create the helmRelease pod:

helm upgrade -i helm-operator fluxcd/helm-operator --wait \
  --namespace fluxcd \
  --set helm.versions=v3

Retrieve the SSH Key:

fluxctl identity --k8s-fwd-ns fluxcd

and add it to your github repository under deploy keys.

You will want to give this key write access so that flux can update your helmReleases when it detects a new version of your image.

Step 4: HelmReleases

Now that flux is all set up, it's time to get to the good stuff. As I mentioned previously the HelmReleases are yaml files that flux will use to deploy an image.

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: <app name -- name of the application you're deploying>
  namespace: default
  annotations:
    flux.weave.works/automated: "true"
    flux.weave.works/tag.chart-image: glob:dev-*
spec:
  releaseName: <app name>
  helmVersion: v3
  chart:
    git: <github repo from step 1>
    path: <the directory where your charts live -- ex. charts/app-name >
    ref: <github branch set in step 2>
  values:
    image: <image endpoint of your application>
    replicaCount: 1
    hpa:
      enabled: true
      maxReplicas: 3
      cpu: 1
    extraEnvs:
      ## This is where you can add any public env variables
      username: oliver_2020
      dbServer: 11.0.1.213 
    envFrom:
      secretRef:
        ## The Secret Name for private env variables (more on that later)
        name: mysecrets

Each HelmRelease you have will need a corresponding Helm Chart. Helm has some great documentation on building custom Helm Charts + Templating, but I'll add the basics.

Step 5: Helm Charts

If kubernetes is the pilot, Helm is the steering wheel. Most Helm Charts contain the following components:

The Chart.yaml file is almost like the 'title page' of a helm chart. It consists of the chart name and version information.

apiVersion: v1
kind: application
description: A Super Awesome GraphQL API
name: graphql
version: 1.0.0

The values.yaml file is where you will define all the variables you will need for your chart.

image: <image endpoint of your application>
    replicaCount: 1
    hpa:
      enabled: true
      maxReplicas: 3
      cpu: 1
    extraEnvs:
      username: oliver_2020

These variables are then consumed by the various template files.

A common template file is the deployment template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
  labels:
    app: {{ template "app.name" . }}
    chart: {{ template "app.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "app.name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "app.name" . }}
        release: {{ .Release.Name }}
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image }}"
          imagePullPolicy: {{ .Values.imagePullPolicy }}
          env:
            - name: publicEnvKey
              value: {{ .Values.extraEnv.key | quote }}
            - name: secretEnvKey
              valueFrom:
                secretKeyRef:
                  key: dbPassword
                  name: mysecrets
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}
      volumes:
      - name: data
        emptyDir: {}

You will have access to variables defined throughout your chart using the templating syntax

.Values
.Release
.Chart

However, if you need to change the structure of a value, for example to remove or add a hyphen, you can add a helper.tpl file. Template helpers are also useful for generating values like timestamps, where it wouldn't make sense to put it in your values.yaml file because it's not really integral to the chart.

Step 6: Template Helpers

Variables are defined in a helper file with the define action

{{- define "mychart.labels" }}
  labels:
    generator: helm
    date: {{ now | htmlDate }}
{{- end }}

You can then call this value in any of your template files with the template action

{{ template "mychart.labels" }}

which the template engine will read as the current date.

For a thorough reference on declaring custom values see the helm named templates guide

Step 7: Sealed Secrets

If you have any private env variables you want to use, it's good practice to store them in a secret. Kubernetes has a native way to store secrets but the values are only base64 encoded which isn't the safest strategy so I've been using a tool called sealed-secrets which allows you to create SealedSecrets resources that you can then store in GitHub because they can only be decrypted by the controller that created them.

A SealedSecret resource looks something like this:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecrets
  namespace: default
spec:
  encryptedData:
    dbPassword: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

and can be referenced in your helmRelease by the secret name

envFrom:
      secretRef:
        name: mysecrets

To begin, install the Stable repository:

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

Update your Helm repositories:

helm repo update

and then install sealed-secrets from the stable repository:

helm install --namespace kube-system stable/sealed-secrets --generate-name

Once it's installed you can apply the sealed-secrets controller to you cluster:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.6/controller.yaml

and then fetch the generated public key (this can be committed to GitHub) using the generated sealed secrets pod name:

kubeseal --fetch-cert \
  --controller-namespace=kube-system \
  --controller-name=sealed-secrets-230493143 \
  > pub-cert.pem

you can grab the pod name with kubectl get pods -n kube-system

Step 8: Creating a SealedSecret Resource

To create a Secret you will first generate it with kubectl and then ecrypt it with kubeseal

kubectl -n <env namespace> create secret generic <secretname> \
--from-literal=<key>=<value> \
--from-literal=<key>=<value> \
--dry-run \
-o json > <secretname>.json

the dry run flag will create the json file with base64 encrypted values, but will not create an actual kubernetes secret on your cluster.

To encrypt it with kubeseal you will pass in the location of the public key from the previous step, the name of the json file you just created, and the filename you want to use for your secret

kubeseal --format=yaml --cert=pub-cert.pem < secretname.json > secretname.yaml

This process will generate a custom SealedSecret resource that contains encrypted credentials that can only be unsealed by the controller that created it.

Once you have your SealedSecret resource you can delete or .gitignore the generated json file. When you commit the pub-cert file and the new SealedSecrets resource to GitHub flux will apply the secret to your cluster and then the controller will unseal it into a native kubernetes secret to be read by your cluster.

Step 9: Push To Git

Once your helm releases and corresponding helm charts are all set you can go ahead and push your commits to the GitHub repository from step 1.

Once your commits are merged or pushed to the branch you specified in step 2 flux will pick up the changes and deploy your first helm release.

Flux generally takes a minute or two to sync, but you can force a sync manually with

fluxctl sync --k8s-fwd-ns fluxcd

replacing fluxcd with whatever namespace you chose for your flux pod in step 2.

Step 10. TroubleShooting

If you pushed your commits, waited for flux to sync, and still aren't seeing the pod with the app-name from your helm release there are a few ways you can troubleshoot.

Get a list of all pods with kubectl get pods --all-namespaces and copy the name of the flux pod. Then run kubectl logs <fluxpodname> -n fluxcd

You should be able to see the logs of flux trying to clone the repo, and polling for updates, and can check for any errors.

Another strategy is kubectl get hr which will get all helm release instances, and will often show where the release is in the deployment process and any errors that came up.

Lastly, I always find it helpful to describe the resources and make sure they are configured as expected. You can describe any kubernetes resource with kubectl describe <resourcetype> <resourcename>

Conclusion

This is only one of many ways to configure a continuous delivery pipeline for your kubernetes pipeline. Flux is a great tool because it integrates well with Helm, and is a relatively straight forward way to automate the deployment process. Helm recently updated to version 3 and flux support for Helm v3 is still in beta mode, thus the somewhat confusing/outdated documentation, but the team at Weaveworks has been very responsive on their slack channel at https://cloud-native.slack.com/ and hopefully this walkthrough can help navigate some of the complexity as well.

Have questions, feedback, pictures of your pets? Don't hesitate to comment down below and happy coding!