In this post, I will jump right into how you can "git" going with GitOps by enabling the FluxCD AKS Extension on your Azure Kubernetes Service (AKS) and using a tool called Kustomize to help with Kubernetes configuration management.
We'll deploy my new favorite demo app, AKS Store Demo to our AKS cluster and then make some changes to the application and see how FluxCD handles them.
Before you begin, you need to have the following:
Use the following Azure CLI commands to create a new resource group and AKS cluster.
RG_NAME=rg-gitops AKS_NAME=aks-gitops LOC_NAME=westus3 az group create -n $RG_NAME -l $LOC_NAME az aks create -n $AKS_NAME -g $RG_NAME
When the cluster is created, run the following command to connect to the cluster.
az aks get-credentials -n $AKS_NAME -g $RG_NAME
AKS extensions are Microsoft-maintained packages that can be installed using Azure Resource Manager to enable additional functionality on your AKS cluster. In a way, AKS extensions are like AKS add-ons. However, AKS extensions are offered via a different Azure resource provider and therefore offers the flexibility to be installed on a variety of Kubernetes clusters including Azure Arc enabled Kubernetes clusters. Take a look at this article by @JorgeArteiro for more information on the differences between AKS extensions and AKS add-ons.
FluxCD is a GitOps tool developed by Weaveworks that allows you to implement continuous and progressive delivery of your applications on Kubernetes. It is a CNCF graduated project that offers a set of controllers to monitor Git repositories and reconciles the cluster's actual state with the desired state defined by manifests committed in the repo.
Before we can install the FluxCD AKS extension using Azure CLI, you need to make sure that you have the proper extensions installed.
Run the following command to install the Azure CLI extensions for Kubernetes.
# enable new features for aks az extension add --name aks-preview # enable aks extension installations az extension add --name k8s-extension # enable aks extension configuration az extension add --name k8s-configuration
With the Azure CLI extensions for Kubernetes installed, you can now install the FluxCD AKS extension using the following command.
az k8s-extension create \ --cluster-name $AKS_NAME \ --resource-group $RG_NAME \ --cluster-type managedClusters \ --extension-type microsoft.flux \ --name aks-store-demo
This command will install the FluxCD AKS extension on your AKS cluster. It is equivalent to running the
flux installFlux CLI command.
The first parameters are self-explanatory... we are using Azure Resource Manager to install the extension so you need to pass in the AKS cluster name and resource group.
The parameters you need to be aware of here are
--extension-type. You need to specify
microsoft.flux respectively which tells the Azure resource provider that you are installing the FluxCD extension on a managed AKS cluster. Getting these last parameters wrong will result in an error.
With the extension installed, you can run the following Flux CLI command to get the status of the installation.
Flux installs many new Custom Resource Definitions (CRDs) in the cluster. These CRDs are how you interact with FluxCD. You can run the following command to see all the new CRDs.
kubectl get crds | grep flux
Next, we need to connect FluxCD to a Git repository, but there's some prep work we need to do first.
A successful GitOps implementation hinges on how well you structure your Git repository and how well you manage processes and workflows. GitOps tools are just that; they're tools, you need to do some work up front like determining file structure, branching strategy, branch protections, and workflows to ensure that you are setting yourself up for success.
The first thing we need to do is decide on how we want to package and manage Kubernetes manifests for different environments. There's a tool called Helm which @StevenMurawski covered here, and another tool called Kustomize which I personally like to use. Both tools are great for packaging and managing Kubernetes manifests and also supported by FluxCD, but I prefer Kustomize because it is pretty easy to use and built into
Kustomize is not a templating engine like Helm, it is more like a patching engine. It allows you to create a base set of Kubernetes manifests and then patch them with environment specific configurations. These environmental configurations are known as "overlays". So you can have a
dev overlay, a
prod overlay, a
staging overlay, etc. and use them to patch the base manifests with environment specific configurations.
Using the GitHub CLI, log into GitHub.
gh auth login
Next fork and clone the aks-store-demo-manifests repository that I created for this tutorial.
gh repo fork https://github.com/pauldotyu/aks-store-demo-manifests.git --clone # change into the repo directory cd aks-store-demo-manifests
All of our manifests currently sit a the root of the repo so we need to create a
base directory so that we can create overlays for different environments.
Run the following command to create a
Copy all the manifests from the root of the repo into the
base directory using the following command:
mv *.yaml base # change into the base directory cd base
We should now have a
base directory with all the manifests in it. Next we need to create our initial
Using the Kustomize CLI, run the following command to create the base
kustomize create --autodetect # view the kustomization.yaml file cat kustomization.yaml
Notice in the
kustomization.yaml file that all of our manifest files were added as resources that Kustomize will patch.
Next, we need to create an overlay for our
We need to navigate back to the root of the repo directory and create a
dev overlay directory.
# navigate back to the root of the repo cd ../ mkdir -p overlays/dev cd overlays/dev
We want our dev deployment to deploy to a new namespace so we'll create a new manifest to create one. Run the following command to create a new manifest file.
kubectl create namespace store-dev --dry-run=client -o yaml > namespace.yaml
kustomization.yaml file for our
dev overlay using the following command.
kustomize create --resources namespace.yaml,./../../base --namespace store-dev # view the kustomization.yaml file cat kustomization.yaml
kustomization.yaml file, we see that it will include our new
namespace.yaml file and all manifests in the
base directory. We also have a
namespace: store-dev entry in the kustomization and this is what instructs Kustomize to patch and add the
store-dev namespace to the base manifests.
Here is what the directory structure of the repo should look like:
. ├── README.md ├── base │ ├── kustomization.yaml │ ├── makeline-service.yaml │ ├── mongodb.yaml │ ├── order-service.yaml │ ├── product-service.yaml │ ├── rabbitmq.yaml │ ├── store-admin.yaml │ ├── store-front.yaml │ ├── virtual-customer.yaml │ └── virtual-worker.yaml └── overlays └── dev ├── kustomization.yaml └── namespace.yaml
This is good enough for now.
Let's test the
dev overlay by running the following command from the
Notice how all the manifests are patched with the
store-dev namespace and output to the console. This is how Kustomize works. It patches the base manifests with environment specific configurations and outputs the patched manifests to the console.
To deploy these patched manifests, you would run a command like this.
kustomize build | kubectl apply -f -
Kustomize is also built into
kubectl so you can run the following command to apply the manifests to your cluster.
kubectl apply -k .
Hopefully you didn't run the commands above. If you did, no sweat 😅 you can delete this deployment with either of these commands.
# using kustomize kustomize build | kubectl delete -f - # or using kubectl kubectl delete -k .
Remember that Git stuff we talked about earlier? Well, let's put Git to use and commit and push our changes the GitHub repo.
Run the following commands to commit and push your changes to GitHub.
# make sure we are back at the root of the repo cd ../../ # add all changes git add . # commit changes git commit -m 'refactor: add kustomize base and dev overlay' # push changes git push
With the repo prepped, we can now connect FluxCD to our GitHub repo. FluxCD will monitor the repo for changes and reconcile the cluster with the desired state defined in the repo.
Run the following command to get the GitHub HTTP URL for your repo.
GH_REPO_URL=$(gh repo view --json url | jq .url -r)
Using Azure CLI again, let's configure the FluxCD AKS extension to connect to our GitHub repo:
az k8s-configuration flux create \ --cluster-name $AKS_NAME \ --resource-group $RG_NAME \ --cluster-type managedClusters \ --name aks-store-demo \ --url $GH_REPO_URL \ --branch main \ --kustomization name=dev path=./overlays/dev \ --namespace flux-system
This command is equivalent to running the
flux create sourceand
flux create kustomizationFlux CLI commands.
Here, we are telling the FluxCD AKS extension to connect to our GitHub repo and monitor the
main branch for changes.
The Azure resource name is set to
aks-store-demo and we pass in a kustomization name of
dev. These two values will be used to create the
Kustomization resources in the cluster.
We also tell the Flux resource where look for our manifests by passing it
Lastly, we tell Flux to create new Flux
Kustomization resources in the
flux-system namespace. You can change this to whatever namespace you want. I used
flux-system for simplicity.
If you run the following Flux CLI commands you should see some resources created.
flux get source git flux get kustomization
If all went well, you should see your pods coming online. Let's watch for them:
kubectl get pods -n store-dev -w
Once you see all the pods running, you can exit the watch by pressing
Let's test the application by grabbing the public IP address of the
kubectl get svc/store-front -n store-dev
Open a browser and navigate to the IP address. You should see the AKS Store Demo application running 🚀
Great, we have successfully deployed our application using GitOps! Now what? Well, let's make some changes to the application and see how FluxCD handles them.
With the FluxCD AKS extension installed and connected to our GitHub repo, we can now make changes by simply editing the kubernetes manifests and committing/pushing the changes back to the remote repo. At this point, it's all about Git workflows and processes.
Let's make a small change to the dev overlay
kustomization.yaml file. Let's say we want to change the name of the namespace from
store-dev to just
overlays/dev/kustomization.yaml file and change the
namespace value from
dev. Optionally, you can run the following
sed command to make the change.
# make sure you are in the root of the repo sed -i -e 's/store-dev/dev/g' overlays/dev/kustomization.yaml
Compare the changes.
git diff overlays/dev/kustomization.yaml
Add the change, commit, and push to GitHub.
git add overlays/dev/kustomization.yaml git commit -m 'refactor: change namespace to dev' git push
I'm pushing directly to the
main branch for simplicity, but you should be using a branch and pull request workflow.
Using the Flux CLI, you can force FluxCD to reconcile the cluster with the desired state defined in the repo.
flux reconcile kustomization aks-store-demo-dev --with-source
After a minute or two you should see the pods coming online in the new
dev namespace. This is FluxCD reconciling the cluster with the desired state defined in the repo.
You can check on the pods using the following command.
kubectl get pods -n dev # type CTRL+C to exit watch
Here's some tips when it comes to monitoring and troubleshooting Flux resources is to use the Flux CLI. You can use some of these basic commands to get information about Flux and its resources:
# check the status of the flux installation flux check # get info about the GitRepository resource flux get source git aks-store-demo-dev -n flux-system # get info about the Kustomization resource flux get kustomization aks-store-demo-dev -n flux-system # view event logs from the flux controllers flux events # view logs from the flux controllers flux log # view stats of the flux controllers flux stats
If you are a little weary about an agent in your Kubernetes cluster doing all this work without your knowledge, don't worry, Flux also allows you to configure notifications for events. You can configure notifications for Slack, Discord, Microsoft Teams, and more. Check out the Flux docs for more information.
In this post, we learned how to enable the FluxCD AKS extension on Azure Kubernetes Service (AKS) and how to deploy applications using a GitOps approach. We did all this using the Azure CLI and the Flux CLI, but you can also use the Azure Portal to install the AKS extension in a "ClickOps" manner 😁
GitOps is a powerful way to do Continuous Delivery of your applications on Kubernetes. But remember, a successful GitOps implementation requires a bit of planning and prep work. You need to think about how you want to manage processes and workflows using a tool like Git. GitOps is not a "silver bullet", it is a way of doing things. It is a way of doing Continuous Delivery of your applications on Kubernetes.
Now, you might be thinking... "Why FluxCD and not ArgoCD?" Well, I will be doing a follow up post on ArgoCD, so stay tuned for that.