Setting Up a Basic Prometheus Deployment with CDK8s

I was genuinely surprised by the scarcity of CDK8s examples available. Consequently, I decided to pitch in and share what I've learned by presenting how I setup a basic Prometheus deployment using CDK8s.

What is CDK8s i hear you ask?

CDK8s is a development framework tailored for Kubernetes. It enables users to define Kubernetes manifests using a library of constructs in their preferred programming language. In essence, CDK8s is akin to AWS CDK but designed for Kubernetes.

Prerequisites

Before we dive into the tutorial, make sure you have the following prerequisites:

• Basic understanding of Kubernetes and Prometheus
• CDK8s CLI installed
• kubectl installed
• A Running Kubernetes cluster (Minikube works well)

1. Initialise your CDK8s application

Begin by setting up your CDK8s app using the init command. While I'll be using TypeScript, there are alternative options available.

cdk8s init typescript-app

This will create a folder structure which looks something like:

my-cdk8s-app/
├── imports/
├── dist/
├── cdk8s.yaml
├── package.json
├── tsconfig.json
└── main.ts

The key file here is main.ts, this contains our CDK8s chart. Charts are just a container for your CDK8s code, they are synthesised into a single K8s manifest.

2. Add imports for necessary constructs

Import the required constructs at the top of your main.ts file to abstract elements like configmaps, volumes, deployments, etc.

main.ts

import { 
  ClusterRole, 
  ApiResource, 
  NonApiResource, 
  ServiceAccount, 
  Deployment, 
  Volume,
  ConfigMap } 
from 'cdk8s-plus-25'

3. Create a namespace for Prometheus and configure the chart

Run the following command in your terminal to create a namespace for Prometheus:

kubectl create namespace monitoring

Modify your main.ts file to ensure all API objects use this namespace:

new MyChart(app, 'CDK8s-Prometheus', { namespace: 'monitoring' })

4. Configure RBAC

For Prometheus to access the Kubernetes API, create a service account, cluster role, and role binding:

    const serviceAccount = new ServiceAccount(this, 'service-account')
    const clusterRole = new ClusterRole(this, 'cluster-role')

    clusterRole.allowRead(ApiResource.NODES) 
    clusterRole.allowRead(ApiResource.SERVICES)
    clusterRole.allowRead(ApiResource.ENDPOINTS)
    clusterRole.allowRead(ApiResource.PODS)
    clusterRole.allowRead(ApiResource.INGRESSES)
    clusterRole.allowGet(NonApiResource.of('/metrics'))


    clusterRole.bindInNamespace('monitoring', serviceAccount)

5. Create a file for Prometheus configuration

Add a prometheus.yml file to the root directory of your project with the following content. This is a simple setup where Prometheus scrape its own metrics

prometheus.yml

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
        - localhost:9090

6. Create volumes for storage and config

Define two volumes to be mounted in the pods later—one for Prometheus config and another for storing metrics data:

    const configMap = new ConfigMap(this, 'configmap')
    configMap.addFile(join(__dirname + '/prometheus.yml'))

    const configVolume = Volume.fromConfigMap(this, 'config-volume', configMap)
    const dataVolume = Volume.fromEmptyDir(this, 'data-volume', 'data')

7. Create our deployment

Specify the deployment details for Prometheus in your main.ts file:

const deployment = new Deployment(this, 'deployment', {

      containers: [ 
        {
          name: 'prometheus',
          image: 'prom/prometheus',
          args: ['--config.file=/etc/prometheus/prometheus.yml'],
          portNumber: 9090,
          securityContext: {ensureNonRoot: false}

        }
      ]
    })

    deployment.containers[0].mount('/etc/prometheus/', configVolume)
    deployment.containers[0].mount('/data/', dataVolume)

8. Deploy your application

With everything set up, deploy your Prometheus instance:

Run:

kubectl apply -f dist/

Test your Prometheus deployment by accessing the UI using port forwarding:

kubectl port-forward <POD NAME> 8080:9090 -n=monitoring

Access the UI via http://localhost:8080

Congratulations! You've successfully set up a basic Prometheus deployment using CDK8s. From here, you can explore further customisation and optimisation based on your specific needs.