OpenTelemetry is a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for understanding, analyzing and debugging purposes.
OTel also promotes the concept of Bring Your Own (BYO) storage backend (eg: elastic stack, AppDynamics, Dynatrace ..) and BYO query (eg: prometheus) by providing common intermediate framework for instrumentation and data collection/transport/transform with OTLP.
OTel kubernetes operator
This operator provides 2 custom resources:
- OTel collector for collection/transform and transport telemetry data
- Auto-instrumentation through init container pattern for supported languages (Python, NodeJS and Java at the moment)
Modes of deployment
The operator(collector CR) can be deployed as:
- Deployment (default)
- DaemonSet
- SideCar injection Note: Multiple modes can co-exist based on the architecture (eg: side-car collectors pushing to a central Deployment aggregator) Official documentation
Installation
# Pre-req: kubernetes cluster with cert-manager enabled
# Deployment based
$ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
namespace/opentelemetry-operator-system created
customresourcedefinition.apiextensions.k8s.io/instrumentations.opentelemetry.io created
...
deployment.apps/opentelemetry-operator-controller-manager created
$ kubectl get crds | grep opentel
instrumentations.opentelemetry.io 2022-11-14T06:14:32Z
opentelemetrycollectors.opentelemetry.io 2022-11-14T06:14:32Z
$ kubectl get deployments.apps -n opentelemetry-operator-system
NAME READY UP-TO-DATE AVAILABLE AGE
opentelemetry-operator-controller-manager 1/1 1 1 56s
# Create a collector like below
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-col
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
logging:
service:
pipelines:
traces:
receivers: [otlp]
processors: []
exporters: [logging]
EOF
opentelemetrycollector.opentelemetry.io/otel-col created
Auto-instrumentation
- Create an instrumentation object
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
EOF
instrumentation.opentelemetry.io/my-instrumentation created
$ kubectl get otelinst
NAME AGE ENDPOINT SAMPLER SAMPLER ARG
my-instrumentation 35s http://otel-collector:4317 parentbased_traceidratio 0.25
ku
- Enabling instrumentation is done through annotations
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "true" # or inject-java or inject-nodejs
Possible values are true, false(no instrumentation), my-instrumentation (custom instrumentation) or my-namespace/my-instrumentation (instrumentation defined in my-namespace).
Further tuning on which containers to instrument can be done with annotation instrumentation.opentelemetry.io/container-names: a,b,c...
When side-car based instrumentation is to be done, use the annotation sidecar.opentelemetry.io/inject: "true"
To use 3rd-party instrumentation, check this link
Demo
My partially working gist so-far is here
$ kubectl apply -f .
opentelemetrycollector.opentelemetry.io/sidecar-collector created
deployment.apps/flask-example created
instrumentation.opentelemetry.io/python-instrumentation created
service/flask-example created
$ kubectl get opentelemetrycollectors.opentelemetry.io,instrumentations.opentelemetry.io
NAME MODE VERSION AGE
opentelemetrycollector.opentelemetry.io/sidecar-collector sidecar 0.63.1 80s
NAME AGE ENDPOINT SAMPLER SAMPLER ARG
instrumentation.opentelemetry.io/python-instrumentation 79s http://localhost:4317 parentbased_traceidratio 0.25
$ kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
flask-example 1/1 1 1 49s
$ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].image}"
ashoka007/flask-example:0.1 ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.63.1%
$ kubectl get pod flask-example-6795f8f4bf-ffhxl -o jsonpath="{.spec.containers[*].name}"
flask-example otc-container%
# otc-container is the local side-car collector
Here is a full-fledged working demo and a detailed blog - https://isitobservable.io/open-telemetry/how-to-observe-your-kubernetes-cluster-with-opentelemetry
Top comments (0)