How many api keys are stored per day as variables in the Gitlab CI configuration?

When a Scaleway Elements API Key is saved in Gitlab, we face all the security issues of storing credentials outside of the cloud infrastructure: Access, authorization, key rotation, age, destruction, location, etc.

There are 2 common reasons for developers to store GCP credentials in Gitlab CI:

• They use shared runners.
• They use specific/group runners deployed in a Scaleway Kubernetes Kapsule cluster but do not use (or do not know about) Gitlab additional configurations.

The alternative that Gitlab CI proposes for users is to mount a secret volume to the runner pods that are created for each build.

Prerequisites

Install the following tools:

• Scaleway cli
• kubectl
• Helm

And create 2 projects in your Scaleway organization:

• devops
• development

Working with Kapsule

The first step is to create the Kapsule devops cluster and configuring our environment [1].

Generate the API Key for the devops project. You can create one by following the documentation How to generate an API key.

scw init
scw k8s cluster create name=kapsule-devops

Let's create a pool for the runner jobs:

scw k8s pool create cluster-id=$(scw k8s cluster list | grep kapsule-devops | awk '{ print $1 }') name=dev node-type=GP1_XS size=2

Add a taint

kubectl taint nodes gitlab-runner-jobs-dev-reserved=true:NoSchedule --selector=k8s.scaleway.com/pool-name=dev

• Configure kubectl to communicate with the cluster:

scw k8s kubeconfig install kapsule-devops

• Create the namespace for the dev runner:

kubectl create namespace dev

• Create the Kubernetes service account to use for specific runner:

kubectl create serviceaccount --namespace dev app-deployer

• Generate an API Key for the specific runner.

Note: For easier visibility and auditing, I recommend to centrally store API keys in a dedicated project and in an external tools like Vault.

• To allow the specific runner to impersonate the API Key we need to store the credentials in a Kubernetes secret.

kubectl create secret generic dev-api-key --from-file ~/.config/scw/config.yaml -n dev

Assign the API Key to the Gitlab runner

The next step is to mount the secret as a data volume [4].

• Start by installing Helm:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

• Add Gitlab Helm package:

helm repo add gitlab https://charts.gitlab.io

• Configure the runner:

Create the file values.yaml:

imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
  create: true
metrics:
  enabled: true
runners:
  image: ubuntu:18.04
  config: |
   [[runners]]
     [runners.kubernetes]
        [[runners.kubernetes.volumes.secret]]
          name = "dev-api-key"
          mount_path = "/root/.config/scw"
          read_only = true
  locked: true
  pollTimeout: 360
  protected: true
  serviceAccountName: app-deployer
  privileged: false
  secret: dev-runner-tokens
  namespace: dev
  builds:
    cpuRequests: 100m
    memoryRequests: 128Mi
  services:
    cpuRequests: 100m
    memoryRequests: 128Mi
  helpers:
    cpuRequests: 100m
    memoryRequests: 128Mi
  tags: "k8s-dev-runner"
  nodeSelector: 
     k8s.scaleway.com/pool-name: dev
  nodeTolerations:
    - key: "gitlab-runner-jobs-dev-reserved"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

You can find the description of each attribute in the Gitlab runner charts repository [2]

• Get the Gitlab registration token from Project -> Settings -> CI/CD -> Runners in the Setup a specific Runner manually section and create the following secret:

kubectl create secret generic dev-runner-tokens --from-literal=runner-token='' --from-literal=runner-registration-token='<TOKEN>' -n dev

• Install the runner:

helm install -n dev app-dev-runner -f values.yaml gitlab/gitlab-runner

Using the specific runner in Gitlab CI

Create the pipeline .gitlab-ci.yml:

stages:
  - dev

infra:
  stage: dev
  image:
    name: scaleway/cli:v2.3.1
  script: 
    - /scw k8s cluster create name=kapsule-dev
    - /scw k8s pool create cluster-id=$(/scw k8s cluster list | grep kapsule-dev | awk '{ print $1 }') name=apps node-type=DEV1_M size=2
  tags:
    - k8s-dev-runner
  only:
    - main

The job will create a Kapsule cluster in the development project. We can follow the same steps for a production environment.

If you want to change the API Key, you just need to delete and recreate the dev-api-key secret.

Access the Kapsule cluster

You can follow the same steps to connect to the Kapsule Cluster from your Gitlab job.

• Create a secret:

kubectl create secret generic dev-kapsule-config --from-file ~/.kube/config -n dev

• and mount it as a data volume:

        [[runners.kubernetes.volumes.secret]]
          name = "dev-kapsule-config"
          mount_path = "/root/.kube"
          read_only = true

In this example, we used a kubeconfig with admin access. You should use RBAC to restrict access to only specific Kubernetes resources that the runner needs.

Implementing Gitlab Flow

Environment branches with Gitlab Flow is a branching strategy and workflow. Suppose you have additional environments like a pre-production environment and a production environment. Deploy the main branch to your development environment. To deploy to pre-production, create a merge request [3] from the main branch to the pre-prod branch. Go live by merging the pre-prod branch into the production branch.

Add the following script ./utils/autoMergeRequest.sh:

#!/bin/bash

[[ $CI_PROJECT_URL =~ ^https?://[^/]+ ]] && CI_PROJECT_URL="${BASH_REMATCH[0]}/api/v4/projects/"

BODY="{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"Deployment to ${TARGET_BRANCH}\",
    \"assignee_id\":\"${GITLAB_USER_MAILINGLIST_ID}\"
}";

LISTMR=`curl --silent "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests?state=opened" --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}"`;
COUNTBRANCHES=`echo ${LISTMR} | grep -o "\"source_branch\":\"${CI_COMMIT_REF_NAME}\"" | wc -l`;

if [ ${COUNTBRANCHES} -eq "0" ]; then
    curl -X POST "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests" \
        --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}" \
        --header "Content-Type: application/json" \
        --data "${BODY}";

    echo "Opened a new merge request: Deployment to ${TARGET_BRANCH} and assigned to you";
    exit;
fi

echo "No new merge request opened";

It's a common pattern to receive the merge request from a mailing list. Create a Gitlab user with this mailing list and add the variable CI/CD GITLAB_USER_MAILINGLIST_ID with the user ID. You can get the ID using the URL https://gitlab.com/api/v4/users?username=<USERNAME>.

Add the Variable CI/CD PRIVATE_TOKEN, you can create one from Settings > Project Access Tokens.

Example of a gitlab-ci.yml:

stages:
  - build
  - dev
  - preprod
  - prod

build:
  stage: build
  script: 
    - echo 'build'
    - TARGET_BRANCH=main ./utils/autoMergeRequest.sh
  only:
    - /^feature\/*/
    - /^hotfix\/*/

dev:
  stage: dev
  script: 
    - echo 'deploy dev'
    - TARGET_BRANCH=preprod ./utils/autoMergeRequest.sh
  tags:
    - k8s-dev-runner
  only:
    - main

preprod:
  stage: preprod
  script: 
    - echo 'deploy preprod'
    - TARGET_BRANCH=prod ./utils/autoMergeRequest.sh
  tags:
    - k8s-preprod-runner
  only:
    - preprod

prod:
  stage: prod
  script: 
    - echo 'deploy prod'
  tags:
    - k8s-prod-runner
  only:
    - prod

The preprod and prod branches need to be marked as protected. Go to Settings > Repository > Protected branches.

Conclusion

In this post, we created a devops cluster, we mounted the config.yml file to the specific runner, and we ended up deploying our Scaleway Elements and Kubernetes resources in an environment project.

This mechanism guarantees end-to-end security for your API Keys resources in Scaleway Elements.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I've convinced you to remove your API keys from Gitlab CI variables.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

Documentation

[1] https://www.scaleway.com/en/docs/compute/kubernetes/api-cli/creating-managing-kubernetes-lifecycle-cliv2/
[2] https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml
[3] https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
[4] https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod