Patrick Domnick

Posted on Feb 25

Packaging and Uploading a Docker Container and Helm Chart to GitLab using GitLab CI, Operator Framework, Kustomize, and Helmify

#docker #kubernetes

This guide serves as a direct follow-up to Building a Kubernetes Operator with the Operator Framework.
It meticulously walks you through the intricacies of packaging a Docker container and Helm Chart, subsequently uploading them to GitLab using GitLab CI, with a keen emphasis on generating semantic version tags. It's noteworthy that nearly all CI templates can be executed locally with minor adjustments to environment variables and utilizing Docker.

Prerequisites

Before commencing, ensure that you have the following tools installed:

yq: brew install yq
docker: brew install docker
kustomize: brew install kustomize
helm: brew install helm
helmify: brew install arttor/tap/helmify

Release Strategy

Before delving into packaging the Docker Container and Helm Chart, let's discuss the strategy in detail. The aim is to create Semantic Releases on a Pipeline Schedule (weekly) to eliminate the need for manual releases. These releases will trigger a pipeline running on a tag, responsible for releasing the Helm Chart and Docker Image. Since pushing changes directly to production is not an option, a pipeline for testing new features and bug fixes will also be implemented.

The workflow unfolds as follows:

Create a new feature branch (feature/improvement).
Make changes to the code.
Submit a Merge Request to the Main Branch (feat: Improve Code). This triggers a pipeline where a Pre-Release is built.
Merge the changes to the Main branch after thorough testing.
Wait for the weekly release to increment the version number (specified as feat, resulting in a Minor Release).
The new tag will publish a Docker Image and Helm Chart to GitLab.

Build the Docker Image

The Operator SDK conveniently provides a Dockerfile, simplifying the building process. We will utilize Kaniko for its user-friendly approach and the fact that it doesn't require a Docker Socket or any form of Docker in Docker. This process ensures the Docker Image is uploaded to the GitLab Container Registry.

.docker:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.20.1-debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json # Authenticate against Gitlab
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile  --destination $CI_REGISTRY_IMAGE:$TAG # Build and Push the Docker Image

Helm

Before delving into the process, it's essential to decide where to store Helm Charts within GitLab. Two viable options are:

Package Registry
GitLab Pages

While the Package Registry appears ideal, it comes with a drawback - a somewhat unconventional domain and Helm Chart path (https://gitlab.com/api/v4/projects/{projectId}/packages/helm/stable). Alternatively, GitLab Pages introduces its own set of challenges. Since Pages primarily deals with static code, properly indexing every version becomes nearly impossible. This guide, however, will focus on indexing the latest version, providing a glimpse into this method.

Packaging the Helm Chart

The Operator SDK employs Kustomize instead of Helm, necessitating additional steps to arrive at our Helm Chart. The initial step involves generating the bundle containing our manifests.

.bundle:
  stage: build
  image: 
    name: quay.io/operator-framework/operator-sdk:v1.32
    entrypoint: [""]
  script:
    - make bundle # Generate the Bundle directory dynamically
  artifacts:
    paths:
      - bundle

Ensure to run the make bundle command at least once locally. This ensures the creation of necessary files for automation later on. Additionally, add the bundle/* directory to the .gitignore file. Also, remember to update the Docker Image of your Manager to the actual Docker Image (registry.gitlab.com/path/to/operator).

Generate and Upload the Helm Chart

Now that we have the necessary manifests, it's time to transform them into a proper Helm Chart. For this purpose, we will utilize Helmify and a custom Docker Image bundling useful tools for Helm Chart upload. After generating the Helm Chart, YQ will be used to set the version. As noted in the release strategy, there is no distinction between Chart and Docker Version. In other words, the Application Version and Helm Chart Version will always align. In this example, we will publish the Helm Chart to both GitLab Pages and the Package Registry.

.upload:
  stage: helm
  variables:
    HELM_EXPERIMENTAL_OCI: 1
  image: registry.gitlab.com/stammkneipe.dev/operator-packager:latest
  before_script:
    - git config --global --add safe.directory '*'
    - VERSION=$(git describe --tags `git rev-list --tags --max-count=1` 2>/dev/null)
    - if [ -z "$VERSION" ]; then VERSION="0.1.0"; fi
    - if [ -z "$CI_COMMIT_TAG" ]; then VERSION="${VERSION}-${CI_PIPELINE_IID}"; fi
  script:
    - kustomize build config/default | helmify ${CI_PROJECT_NAME} # Generate the Helm Chart with Kustomize and Helmify
    - yq e -P ".version = \"${VERSION}\"" -i ${CI_PROJECT_NAME}/Chart.yaml # Set the Version of the Helm Chart
    - yq e -P ".appVersion = \"${VERSION}\"" -i ${CI_PROJECT_NAME}/Chart.yaml # Set the Version of the Application
    - helm package ${CI_PROJECT_NAME} --destination ./public
    - helm repo add --username gitlab-ci-token --password ${CI_JOB_TOKEN} ${CI_PROJECT_NAME} ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/helm/stable # Only for Package Registry
    - helm plugin install https://github.com/chartmuseum/helm-push # Only for Package Registry
    - helm cm-push ./public/${CI_PROJECT_NAME}-${VERSION}.tgz ${CI_PROJECT_NAME} # Only for Package Registry
    - helm repo index --url https://${CI_PROJECT_NAMESPACE}.gitlab.io/${CI_PROJECT_NAME} . # Only for Gitlab Pages
    - mv index.yaml ./public # Only for Gitlab Pages
  artifacts:
    paths:
      - public # Gitlab Pages

Generating the Actual GitLab CI Jobs

Up until now, we've been creating templates for reuse in actual CI Jobs. Now it's time to generate the full [GitLab CI] configuration.

Stages

For this pipeline, we only need three stages. The build stages will handle building the Docker Image and the manifests for our Helm Chart, while the release stage will exclusively manage the Semantic Release.

stages:
  - build
  - helm
  - release

Feature Branches

Feature or Renovate jobs will only run when a Merge Request is made to your Main Branch:

.feature_renovate:
  variables:
    TAG: $CI_COMMIT_REF_SLUG-$CI_PIPELINE_IID
  rules:
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^renovate/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^fix/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feat/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH

containerize_feature:
  extends:
    - .docker
    - .feature_renovate

bundle_feature:
  extends:
    - .bundle
    - .feature_renovate

upload_feature:
  extends:
    - .upload
    - .feature_renovate

Main Branch

The Main Branch will serve two purposes. Firstly, we want to create a latest build, and on top of that, we want to generate semantic releases. This requires ensuring that semantic releases only run on a schedule, and other jobs do not run on schedule.

.latest:
  variables:
    TAG: latest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"

containerize_latest:
  extends:
    - .docker
    - .latest

bundle_latest:
  extends:
    - .bundle
    - .latest

upload_latest:
  extends:
    - .upload
    - .latest

release_weekly:
  stage: release
  image: registry.gitlab.com/stammkneipe.dev/semantic-release:latest
  script:
    - git config --global --add safe.directory '*'
    - npx -p @semantic-release/changelog -p @semantic-release/exec -p @semantic-release/git -p @semantic-release/gitlab semantic-release
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Version Tag

The last jobs will be used for the actual versioned release and only run on a tag.

.tag:
  variables:
    FULL_IMAGE_NAME: $CI_COMMIT_TAG
  rules:
    - if: $CI_COMMIT_TAG

containerize_tag:
  extends:
    - .docker
    - .tag

bundle_tag:
  extends:
    - .bundle
    - .tag

upload_tag:
  extends:
    - .upload
    - .tag

Creating the Schedule

The only thing left to do is to create a schedule to your liking. Keep in mind that Semantic Release will not create a release when there was no change in your code base. So there is no harm in using a short interval.

Usage

Now you can choose between your latest version inside Pages or use the Registry Package (https://gitlab.com/api/v4/projects/{projectId}/packages/helm/stable) to install your Helm chart.

Conclusion

By following these steps, you've successfully set up a GitLab CI pipeline for packaging and deploying a Docker container and Helm Chart. The integration of semantic versioning ensures a structured release process for your application. This streamlined workflow enhances collaboration and facilitates the continuous delivery of your containerized applications.

DEV Community