As a Kubernetes user, I chose to use the default tooling available, named
Kustomize, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that,
Kustomize is a powerful solution based on concepts of inheritance (
resources) and composition (
But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where
helm is ahead of
Kustomize provides a mechanism to use
remote elements, for both
git under the hood. This solution is perfect, because we can use any
git repository as a
yaml registry for our
But, the downside is
FluxCD does not supports so well, for a lot of good reasons (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote
In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!
For this article, I have set up a cluster using Google Kubernetes Engine (aka GKE) using this documentation and installed
FluxCD connected to a GitLab repository with the following command (extracted from the official documentation).
$ flux bootstrap gitlab \ --owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \ --repository=clusters \ --branch=gke \ --path=fluxcd
NOTE: I don't like to use
default branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of GitLab Flow for environment.
With this, I have a complete Kubernetes Cluster ready for GitOps, using
clusters repository as source-of-truth. Every modification made in this repository will be automatically deployed by
I have set up another
git repository dedicated to
components. It hosts what we usually call
base, because it contains the common core of an application and also many
components to enable some extra features (at infrastructure or application level).
This repository is available in GitLab and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an
helm registry. Because it is managed in
git, I can use branch & tags for management 🚀.
NOTE: Usually, this kind of
yaml repository is only populated by CI from other projects automatically.
I have chosen the well known
podinfo application as an example, and I have developed the
base and some
components associated to it.
. ├── README.md └── podinfo └── base ├── components │ ├── hpa │ │ ├── hpa.yaml │ │ └── kustomization.yaml │ ├── ingress │ │ ├── ingress.yaml │ │ └── kustomization.yaml │ └── redis │ ├── kustomization.yaml │ ├── redis.conf │ └── redis.yaml ├── kustomization.yaml └── podinfo.yaml
This part is the common part of all potential deployment. It contains:
And nothing else… if you need to deploy any other part of the app, you have to use
components, we can find all optional part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.
Here, we have:
hpawith a default preconfigured
Horizontal-Pod-Autoscalerif you want the application to scale up and down automatically
ingressdefinition to expose the application to the outside world (see official documentation)
rediscomponent to enable cache on the PodInfo application with default configuration.
It's interesting to note components can be of any sort,
ingress are focused on infrastructure when
redis is a feature of the application. Without this
redis component, the application is able to work, just without any caching system. This
redis component is here to deploy a
redis server (of course 😅) but also to enable caching at
podinfo level too.
NOTE: If you want a better description of this
component feature, I advise you to discover this article from the official documentation
So, we have a complete, environment agnostic
components committed to our
base repository. Now, It is time to use it!
Back to our cluster, I want to deploy this
podinfo application, but of course I want to apply some extra customizations on it, like:
componentsI want to enable
- Define a domain name for the
- Define the required
tlsconfiguration to the
- Define an alternative registry for the image
So, to do that, I need to have one (or multiple)
kustomization.yaml to define this:
# <clusters-repo>/podinfo-dev/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: podinfo-dev images: - name: podinfo newName: ghcr.io/stefanprodan/podinfo resources: - ../base - ingress components: - ../base/components/redis
ingress modification is in its own file to keep this one simple enough.
But, what is this
../base folder? There is no
base at all in
cluster repository. And you are right… we will use
FluxCD feature available in flux to
base repository at the right location during
reconciliation. To do that, we need to define multiple flux resources.
First, we need a representation of our
bases, which is materialized in flux with:
apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: GitRepository metadata: name: podinfo-base-dev namespace: flux-system spec: interval: 5m ref: branch: podinfo url: ssh://firstname.lastname@example.org/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases secretRef: name: flux-system
NOTE: Here, I have chosen to define
spec.ref.branch: podinfo to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.
Now, we need to create a
flux.GitRepository for manifests in
clusters repository and manifests from the
bases repository created in the step before. To do that, we are going to use the
include feature from FluxCD:
apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: GitRepository metadata: name: podinfo-dev namespace: flux-system spec: interval: 5m include: - repository: name: podinfo-base-dev fromPath: podinfo/base toPath: base ref: branch: gke url: ssh://email@example.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters secretRef: name: flux-system
The important part is the following:
spec: include: - repository: name: podinfo-base-dev # <1> fromPath: podinfo/base # <2> toPath: base # <3>
This means the folder
podinfo/base (2) previously defined in the
flux.GitRepository (1) will be injected in the path
base (3) of the current
flux.GitRepository during FluxCD reconciliation… making all
This one is the standard
flux.Kustomization you would use in any case. This one is just using the composite
flux.GitRepository from the previous step as source (
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1 kind: Kustomization metadata: name: podinfo-dev namespace: flux-system spec: suspend: false interval: 10m0s path: podinfo-dev # file path in the <clusters> repository prune: true sourceRef: kind: GitRepository namespace: flux-system name: podinfo-dev validation: client
❯ kubectl get all -n podinfo-dev NAME READY STATUS RESTARTS AGE pod/redis-869ff7c78b-jjbbk 1/1 Running 0 103m pod/podinfo-6dbf56d6d7-ctxxr 1/1 Running 0 103m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/podinfo ClusterIP 10.43.115.207 <none> 9898/TCP,9999/TCP 103m service/redis ClusterIP 10.43.192.84 <none> 6379/TCP 103m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis 1/1 1 1 103m deployment.apps/podinfo 1/1 1 1 103m NAME DESIRED CURRENT READY AGE replicaset.apps/redis-869ff7c78b 1 1 1 103m replicaset.apps/podinfo-6dbf56d6d7 1 1 1 103m
Now, we want to use again the same
base this time for a production environment.
Because we want something relatively stable, we chose to define
spec.ref.tag to a specific tag in the
bases repository (instead of the
apiVersion: source.toolkit.fluxcd.io/v1beta1 kind: GitRepository metadata: name: podinfo-base-prod namespace: flux-system spec: interval: 5m ref: tag: podinfo-1.0 url: ssh://firstname.lastname@example.org/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases secretRef: name: flux-system
flux.GitRepository are the same.
Because we are in production, we also want to enable different modules, here the
HorizontalPodAutoscaler. So we have a different
kustomization.yaml. Thanks to the full control we have over the
kustomization.yaml per environment, we can modify this only in production 😍.
Once everything is published and FluxCD reconciliation is OK, we have:
❯ kubectl get all -n podinfo-prod NAME READY STATUS RESTARTS AGE pod/redis-869ff7c78b-lnll5 1/1 Running 0 113m pod/podinfo-6dbf56d6d7-k8rjl 1/1 Running 0 113m pod/podinfo-6dbf56d6d7-qldr8 1/1 Running 0 113m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/podinfo ClusterIP 10.43.109.213 <none> 9898/TCP,9999/TCP 113m service/redis ClusterIP 10.43.252.54 <none> 6379/TCP 113m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/redis 1/1 1 1 113m deployment.apps/podinfo 2/2 2 2 113m NAME DESIRED CURRENT READY AGE replicaset.apps/redis-869ff7c78b 1 1 1 113m replicaset.apps/podinfo-6dbf56d6d7 2 2 2 113m NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/podinfo Deployment/podinfo 4%/99% 2 4 2 113m
It was tough, with many
yaml but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (
yaml) registry, like
helm already have 😇.
There is some follow-up improvements to come, like OCI registries (announced in FluxCD 0.32), to replace
flux.GitRepository. We still need this issue to be solved to be able to use it with all features provided by Kustomize.
In a more distant future, some other solution like porch could be a good candidate to simplify again our deployment strategy… but it is currently in
I hope you liked this article, you can find all resources used in GitLab, and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter @davinkevin