What are we doing?
Kubernetes SD configurations allow retrieving scrape targets from Kubernetes' REST API in order to stay synchronized with the cluster state.
I created an image of a Python program that exports metrics using the the Prometheus Python client. Setting the environment variable EXPORTER_PORT will publish metrics to that port.
I deployed this container in pods with labels foo and bar and sent metrics to port 9000 and 8000 respectively. You can see in the below Prometheus config that I target them separately by their label name.
Why would you need to do this?
I found a case where I wanted different scrape configurations per pod in my cluster. Usually, I can set the kubernetes_sd_configs role to service, which discovers a target for each service port for each service (see the docs). However, in this case I had different collection requirements for the pods inside my k8s abstractions.
Example config files
Foo Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: foo-deployment
labels:
app: foo
spec:
selector:
matchLabels:
app: foo
replicas: 2
template:
metadata:
labels:
app: foo
spec:
containers:
- name: python-prometheus-exporter-example
image: ahf90/python-prometheus-exporter-example
imagePullPolicy: Always
ports:
- containerPort: 9000
env:
- name: EXPORTER_PORT
value: '9000'
Bar Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: bar-deployment
labels:
app: bar
spec:
selector:
matchLabels:
app: bar
replicas: 2
template:
metadata:
labels:
app: bar
spec:
containers:
- name: python-prometheus-exporter-example
image: ahf90/python-prometheus-exporter-example
imagePullPolicy: Always
ports:
- containerPort: 8000
env:
- name: EXPORTER_PORT
value: '8000'
Prometheus config.yml
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'foo-targets'
scrape_interval: 10s
metrics_path: '/'
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- default
selectors:
- role: "pod"
label: "app=foo"
- job_name: 'bar-targets'
scrape_interval: 5s
metrics_path: '/'
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- default
selectors:
- role: "pod"
label: "app=bar"
Top comments (0)