DEV Community

Daniel German Rivera for AWS Community Builders

Posted on • Updated on

Smart-Cash Project - GitOps with FluxCD

In a previous article I mentioned the idea behind this project that I named SmartCash. I began building the terraform code for the infrastructure in AWS and the pipeline to deploy it.

In this article, I will introduce FluxCD as a GitOps tool and demonstrate its usage.

GitOps meme, source https://blog.kubesimplify.com/gitops-demystified

Source code

A new release has been created in the smart-cash repository for the project. v1.1.0 version will be used, you can check the repository here.

Additionally, a new repository will be created to store the K8 manifest that will be synced with the EKS cluster using FluxCD, you can view the repo here.

A quick introduction to GitOps

GitOps is an operational model for cloud-native architectures, it relies on a Git repository as the single source of truth. New commits imply infrastructure and application updates.

OpenGitOps group has defined 5 principles, and while I won't delve into them, here, you can read more. If you take a look at those principles you will see that they are, in some sense related to some Kubernetes concepts.

A great book to gain a better understanding of GitOps history and concepts is The Path to GitOps.

In summary, GitOps is centered around using a Git repository for defining and managing both infrastructure and application configurations through a Git-based workflow.

What is FluxCD

FluxCD is an open-source GitOps operator for Kubernetes, you can declaratively define the desired state of your infrastructure and configurations in a Git repository. Flux monitors the repository and applies updates to the Kubernetes cluster when new changes arrive.

Flux started as a monolith but in v2 it was broken up into individual components called GitOps Toolkit, this refers collection of specialized tools, Flux Controllers, composable APIs, and reusable Go packages available under the fluxcd GitHub organization.

Core concepts and toolkit components are described here.

hands-on

Installing FluxCD in the cluster

FluxCD installation can be done by Flux CLI, the most straightforward method can be done by the flux bootstrap command, this deploys the Flux controllers on the K8 cluster and configures them to synchronize the cluster to the Git repository, if the Git Repo doesn't exist, the bootstrap command will create it.

To incorporate FluxCD installation into the pipeline created for Terraform, you can add a new job to the workflow template and perform the following steps:

  1. Validate if flux is installed
  2. Install FluxCD CLI
  3. Get credentials for the EKS cluster
  4. Run flux bootstrap command.

Those steps have been added in a bash script that will be placed in the .github/jobs directory.

#!/bin/bash
aws eks update-kubeconfig --name $CLUSTER_NAME  --region $AWS_REGION

flux_installed=$(kubectl api-resources | grep flux)
if [ -z "$flux_installed" ]; then
  echo "flux is not installed"

  curl -s https://fluxcd.io/install.sh | sudo bash

  flux bootstrap github \
    --owner=$GH_USER_NAME \
    --repository=$FLUX_REPO_NAME \
    --path="clusters/$ENVIRONMENT/$CLUSTER_NAME/bootstrap" \
    --branch=main \
    --personal
else
  echo "flux is installed"
fi
Enter fullscreen mode Exit fullscreen mode

The flux bootstrap github command deploys the Flux controllers on the K8 cluster and configures the controllers to synchronize the Git repo with the cluster. This is done by some K8 manifests that are created and pushed to the repo in the path passed in the command.

It's worth noting that some env variables like FLUX_REPO_NAME, and GH_USER_NAME are used by the bash script, these have been added as GH Workflow Input in the template.

Adding FluxCD bootstrap to the GitHub workflow

The bash script will be executed in the GH workflow template created to deploy the infrastructure, the following job is added to the Workflow template.

  flux-bootstrap:
    runs-on: ubuntu-latest
    needs: terraform-apply  
    steps:
     - name: checkout-repo
       uses: actions/checkout@v4
     - name: configure aws credentials
       # Removed for brevity
     - name: bootstrap flux
       id: bootstrap-flux
       run: ./bootstrap-flux.sh
       working-directory: .github/jobs/
       env: 
        GITHUB_TOKEN: ${{ secrets.GH_TOKEN_FLUX }}
Enter fullscreen mode Exit fullscreen mode

Notice that the GITHUB_TOKEN variable is passed directly in the job, this is to avoid propagating the token in all the jobs, for now, it is just necessary for the Flux bootstrap. PAT is stored as a GitHub Secret in the repo.

Once the workflow is ready you can push it to the repo and see the logs in GH actions, which can looks like this:

bootstrap-logs

You can run flux check command locally to validate the status of the installation(you should have access to the cluster in your local env)

flux-check

If you take a look at the above image you will see that the Source Controller is deployed, Source Controller enables seamless integration of various Git repositories with your Kubernetes cluster. Think of the Source Controller as an interface to connect with GitRepositories, OCIRepository, HelmRepository, and Bucket resources.

✔ source-controller: deployment ready

The bootstrap command will create a flux source and associate it to the repo passed in the command, to validate this you can list the git sources created and you will see the one, for now.

flux get sources git
Enter fullscreen mode Exit fullscreen mode

Flux-git-source

and you can see the K8 CDRs created

kubectl get crds | grep flux

Structuring the Git repository

There are different strategies to structure the GitOps repository, for this scenario, a mono-repo strategy is used and kustomize will be used to manage the K8 manifest for the application.

  • ./clusters/smart-cash-develop: Yaml files that create new FluxCD resources like sources, Kustomizations, and Helm releases.
  • common: Manifests that create common resources for the clusters.
  • app: Manifests that create resources for an application or service.
├── clusters
|   └── smart-cash-develop
|       |── flux-system
|       |── helm-source.yaml
|       |── common-kustomize.yaml
|── common
|   └── ingress-namespace.yaml
└── app1
    |── base
    |   |── deployment.yaml
    |   └── service.yaml
    └── overlays.yaml 
        |── dev 
        |   └── kustomization.yaml
        └── prod
            └── kustomization.yaml

Enter fullscreen mode Exit fullscreen mode

Adding resources to the cluster

Let's create a K8 namespace to be used for an nginx-ingress. The manifest for this can be placed in the common folder. A FluxCD Kustomization can be added to synchronize the contents of this folder with the K8 cluster.

The following is the Flux Kustomization that reconciles the Kubernetes manifests located at the path ./common in the Git repository .

  • Note: This file can be added in clusters/smart-cash-develop folder, FluxCD will automatically create the Kustomization resource because this path was specified in the bootstrap command, and Flux created a Kustomization to synchronize it.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: smartcash-common
  namespace: flux-system
spec:
  interval: 5m
  targetNamespace: default
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: "./kustomize"
  prune: true
Enter fullscreen mode Exit fullscreen mode
  • interval: The period at which the Kustomization is reconciled.
  • sourceRef: refers to the Source object with the required Artifacts, in this case, our GitOps repository.
  • prune:: When is true, if previously applied objects are missing from the current revision, these objects are deleted from the cluster

Once you push the Yaml file to the GitOps repo, Flux will create the resources in the cluster. You can validate running:

kubectl get kustomization -n flux-system

common-kustomization

The previous steps have created the FluxCD Kustomization to sync the common folder with the cluster. Now, a Kustomize file needs to be added to specify which resource to create.

Don't confuse the FluxCD Kustomization file with the K8 configuration management Kustomize. FluxCD will look for the Kustomize file in the common folder.

Let's create and push the following files in the common folder.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ns-nginx-ingress.yaml
Enter fullscreen mode Exit fullscreen mode
apiVersion: v1
kind: Namespace
metadata:
  name: nginx-ingress
Enter fullscreen mode Exit fullscreen mode

You can wait for the flux reconciliation or force it using the following command:

flux reconcile kustomization smartcash-common
Enter fullscreen mode Exit fullscreen mode

If the process was successful you should see the nginx-ingress namespace.

Troubleshooting

To validate the status of the reconciliation you can use the following command:

flux get kustomization smartcash-common
Enter fullscreen mode Exit fullscreen mode

For instance, a mistake in the name of the YAML files caused this error, which was visible in the output of the flux command.

Flux-error

If you want more details you can check the K8 CDRs using:

kubectl describe kustomization smartcash-common -n flux-system 
Enter fullscreen mode Exit fullscreen mode

Creating a Helm release for nginx-ingress

The Flux Helm Controller will be used to install the ingress. The Helm Controller is a Kubernetes operator that enables the management of Helm chart releases.

A FluxCD source for Helm needs to be added. This can be accomplished by using the following manifest, which should be placed in clusters/smart-cash-develop.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: helm-repo-nginx-ingress
  namespace: flux-system
spec:
  interval: 5m0s
  type: oci
  url: oci://ghcr.io/nginxinc/charts
Enter fullscreen mode Exit fullscreen mode

This source fetches the Helm OCI repository oci://ghcr.io/nginxinc/charts every 5 minutes, and the artifact is stored and updated each time new updates are done to the repository.

After creating the Helm source, you can proceed to create the Helm release. This release specifies the chart to install in the cluster, with the chart being fetched from the source already created. The following manifest can be used.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  interval: 10m0s
  chart:
    spec:
      chart: nginx-ingress
      version: 0.17.1
      sourceRef:
        kind: HelmRepository
        name: helm-repo-nginx-ingress
        namespace: flux-system

Enter fullscreen mode Exit fullscreen mode

To delegate the creation of the HelmRelease task to flux, this file can be added to the common folder and in the Kustomize file as well.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ns-nginx-ingress.yaml
- nginx-ingress-helm.yaml
Enter fullscreen mode Exit fullscreen mode

After updating and pushing the files, you can validate the creation of the Helm Release and nginx-ingress resources.

flux get helmreleases -n nginx-ingress
Enter fullscreen mode Exit fullscreen mode

Helm-releases

Up to this point, we've covered the second phase of this project. In the upcoming articles, you'll delve into the implementation of various other tools and continue building the project.

If you have any feedback or suggestions, please feel free to reach out to me on LinkedIn.

Top comments (0)