Deploy with Kustomize, FluxCD and Remote Resources

Introduction

As a Kubernetes user, I chose to use the default tooling available, named Kustomize, to manage my manifests and all modifications I need to do over them for each environment I want to deploy. For that, Kustomize is a powerful solution based on concepts of inheritance (resources) and composition (components).

But one thing I miss the most is the ability to distribute manifests to other people, letting them inherit or compose them as they want. This is one subject where helm is ahead of Kustomize now.

However, Kustomize provides a mechanism to use remote elements, for both resources and components using git under the hood. This solution is perfect, because we can use any git repository as a yaml registry for our reousrces and components 🔥.

But, the downside is FluxCD does not supports so well, for a lot of good reasons (especially when your repositories are private)… In this article, we will see how to bypass this limitation and use remote resources and components, the FluxCD way!

Cluster & FluxCD configuration

In this article, I won't detail so much the cluster & flux configuration… because there is already plenty of documentation and blogposts about that!

For this article, I have set up a cluster using Google Kubernetes Engine (aka GKE) using this documentation and installed FluxCD connected to a GitLab repository with the following command (extracted from the official documentation).

$ flux bootstrap gitlab \
  --owner=davinkevin.fr/articles/flux-and-kustomize-remote-base \
  --repository=clusters \
  --branch=gke \
  --path=fluxcd

NOTE: I don't like to use default branch, so I chose to create a branch for this cluster, so if I have another cluster, I just need another branch 😉. This is some kind of GitLab Flow for environment.

With this, I have a complete Kubernetes Cluster ready for GitOps, using clusters repository as source-of-truth. Every modification made in this repository will be automatically deployed by FluxCD 🤩.

bases repository

I have set up another git repository dedicated to resources and components. It hosts what we usually call base, because it contains the common core of an application and also many components to enable some extra features (at infrastructure or application level).

This repository is available in GitLab and is agnostic of my deployment environment. Again, I only host here the bare definition of my applications, we can compare this to an helm registry. Because it is managed in git, I can use branch & tags for management 🚀.

NOTE: Usually, this kind of yaml repository is only populated by CI from other projects automatically.

PodInfo base publication

I have chosen the well known podinfo application as an example, and I have developed the base and some components associated to it.

.
├── README.md
└── podinfo
    └── base
        ├── components
        │   ├── hpa
        │   │   ├── hpa.yaml
        │   │   └── kustomization.yaml
        │   ├── ingress
        │   │   ├── ingress.yaml
        │   │   └── kustomization.yaml
        │   └── redis
        │       ├── kustomization.yaml
        │       ├── redis.conf
        │       └── redis.yaml
        ├── kustomization.yaml
        └── podinfo.yaml

Core PodInfo

This part is the common part of all potential deployment. It contains:

• The podinfo Kubernetes Service
• The podinfo Kubernetes Deployment

And nothing else… if you need to deploy any other part of the app, you have to use components 👇.

Components

In this components, we can find all optional part of our application. As an end-user, I need to declare those I want to use, because by default, they are not applied at all in the common core.

Here, we have:

• hpa with a default preconfigured Horizontal-Pod-Autoscaler if you want the application to scale up and down automatically
• ingress definition to expose the application to the outside world (see official documentation)
• redis component to enable cache on the PodInfo application with default configuration.

It's interesting to note components can be of any sort, hpa and ingress are focused on infrastructure when redis is a feature of the application. Without this redis component, the application is able to work, just without any caching system. This redis component is here to deploy a redis server (of course 😅) but also to enable caching at podinfo level too.

NOTE: If you want a better description of this component feature, I advise you to discover this article from the official documentation

So, we have a complete, environment agnostic resource and components committed to our base repository. Now, It is time to use it!

Publish PodInfo to dev

Back to our cluster, I want to deploy this podinfo application, but of course I want to apply some extra customizations on it, like:

• Define components I want to enable
• Define a domain name for the ingress definition
• Define the required tls configuration to the ingress
• Define an alternative registry for the image

So, to do that, I need to have one (or multiple) kustomization.yaml to define this:

# <clusters-repo>/podinfo-dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: podinfo-dev

images:
  - name: podinfo
    newName: ghcr.io/stefanprodan/podinfo

resources:
  - ../base
  - ingress

components:
  - ../base/components/redis

NOTE: The ingress modification is in its own file to keep this one simple enough.

But, what is this ../base folder? There is no base at all in cluster repository. And you are right… we will use FluxCD feature available in flux to include the base repository at the right location during reconciliation. To do that, we need to define multiple flux resources.

podinfo-base flux repository

First, we need a representation of our flux.GitRepository for bases, which is materialized in flux with:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-base-dev
  namespace: flux-system
spec:
  interval: 5m
  ref:
    branch: podinfo
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
  secretRef:
    name: flux-system

NOTE: Here, I have chosen to define spec.ref.branch: podinfo to follow every commit made to this branch. It is, for development, a continuous deployment system 🚀.

podinfo flux repository

Now, we need to create a flux.GitRepository for manifests in clusters repository and manifests from the bases repository created in the step before. To do that, we are going to use the include feature from FluxCD:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-dev
  namespace: flux-system
spec:
  interval: 5m
  include:
    - repository:
        name: podinfo-base-dev
      fromPath: podinfo/base
      toPath: base
  ref:
    branch: gke
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/clusters
  secretRef:
    name: flux-system

The important part is the following:

spec:
  include:
    - repository:
        name: podinfo-base-dev # <1>
      fromPath: podinfo/base # <2>
      toPath: base # <3>

This means the folder podinfo/base (2) previously defined in the podinfo-base-dev flux.GitRepository (1) will be injected in the path base (3) of the current podinfo-dev flux.GitRepository during FluxCD reconciliation… making all resources and components available.

podinfo-dev flux Kustomization

This one is the standard flux.Kustomization you would use in any case. This one is just using the composite flux.GitRepository from the previous step as source (spec.sourceRef):

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: podinfo-dev
  namespace: flux-system
spec:
  suspend: false
  interval: 10m0s
  path: podinfo-dev # file path in the <clusters> repository  
  prune: true
  sourceRef:
    kind: GitRepository
    namespace: flux-system
    name: podinfo-dev
  validation: client

If we deploy all those files (you can put them in the same one), you will see your application deployed by FluxCD with your provided custom configuration.

❯ kubectl get all -n podinfo-dev
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-869ff7c78b-jjbbk     1/1     Running   0          103m
pod/podinfo-6dbf56d6d7-ctxxr   1/1     Running   0          103m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.43.115.207   <none>        9898/TCP,9999/TCP   103m
service/redis     ClusterIP   10.43.192.84    <none>        6379/TCP            103m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis     1/1     1            1           103m
deployment.apps/podinfo   1/1     1            1           103m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-869ff7c78b     1         1         1       103m
replicaset.apps/podinfo-6dbf56d6d7   1         1         1       103m

Publish PodInfo to prod

Now, we want to use again the same base this time for a production environment.

Because we want something relatively stable, we chose to define spec.ref.tag to a specific tag in the bases repository (instead of the podinfo branch).

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: podinfo-base-prod
  namespace: flux-system
spec:
  interval: 5m
  ref:
    tag: podinfo-1.0
  url: ssh://git@gitlab.com/davinkevin.fr/articles/flux-and-kustomize-remote-base/bases
  secretRef:
    name: flux-system

All others flux.Kustomization and flux.GitRepository are the same.

Because we are in production, we also want to enable different modules, here the HorizontalPodAutoscaler. So we have a different kustomization.yaml. Thanks to the full control we have over the kustomization.yaml per environment, we can modify this only in production 😍.

Once everything is published and FluxCD reconciliation is OK, we have:

❯ kubectl get all -n podinfo-prod
NAME                           READY   STATUS    RESTARTS   AGE
pod/redis-869ff7c78b-lnll5     1/1     Running   0          113m
pod/podinfo-6dbf56d6d7-k8rjl   1/1     Running   0          113m
pod/podinfo-6dbf56d6d7-qldr8   1/1     Running   0          113m

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/podinfo   ClusterIP   10.43.109.213   <none>        9898/TCP,9999/TCP   113m
service/redis     ClusterIP   10.43.252.54    <none>        6379/TCP            113m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/redis     1/1     1            1           113m
deployment.apps/podinfo   2/2     2            2           113m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/redis-869ff7c78b     1         1         1       113m
replicaset.apps/podinfo-6dbf56d6d7   2         2         2       113m

NAME                                          REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/podinfo   Deployment/podinfo   4%/99%    2         4         2          113m

Conclusion

It was tough, with many yaml but we have achieved our original goal. Being able to publish bare manifests in a central place and allow other team / people to consume them instantaneously with all features available in Kustomize.
We are then able to enjoy all the feature of a GitOps model with a dedicated (yaml) registry, like helm already have 😇.

There is some follow-up improvements to come, like OCI registries (announced in FluxCD 0.32), to replace flux.GitRepository. We still need this issue to be solved to be able to use it with all features provided by Kustomize.

In a more distant future, some other solution like porch could be a good candidate to simplify again our deployment strategy… but it is currently in alpha stage.

I hope you liked this article, you can find all resources used in GitLab, and you can reproduce it in your own cluster! If you have any questions, don't hesitate to comment or ping me on twitter @davinkevin