Logging demo with OTEL Collector, CloudWatch and Grafana

Hello 👋, in this post, we would be using the OTEL Demo App with OpenTelemetry Collector and export the logs from the microservices in the app to AWS CloudWatch.

Let's get started!!! We can create a separate namespace in the kubernetes cluster with kubectl create ns otelcol-logs-demo.

And then create a secret that holds the AWS credentials.

kubectl create secret generic aws-credentials \
    --from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
    --from-literal=AWS_SECRET_ACCESS_KEY=<aws-secret-access-key> \
    -n otelcol-logs-demo

We would be deploying open telemetry demo app, otel collector as well as Grafana with a single helm chart. The chart comes with other components too, we could disable those that we do not need for this lab. Our helm values would like below.

$ cat values.yaml 
opentelemetry-collector:
  config:
    exporters:
      awscloudwatchlogs:
        log_group_name: otel-logs-demo
        log_stream_name: otel-demo-app
        region: ap-south-2
    service:
      pipelines:
        logs:
          exporters:
            - awscloudwatchlogs
        metrics: {}
        traces: {}
  extraEnvsFrom:
    - secretRef:
        name: aws-credentials
prometheus:
  enabled: false
jaeger:
  enabled: false
opensearch:
  enabled: false

In the values above, we have the cloudwatchlogs exporter section where we specify the region, log group and stream names. We are only setting up logs in the pipeline as we are only dealing with that for this demo. We are then setting the AWS credentials via env vars so that the collector could authenticate and send logs to cloud watch. Then, finally we are disabling certain applications which we do not need for this exercise.

Let's install the chart.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install my-otel-demo open-telemetry/opentelemetry-demo -n otelcol-logs-demo -f values.yaml

Head over to CloudWatch logs in the mentioned region and you should be able to see the log group. And, inside the log group we should have the stream.

We should be able to view the logs, for ex. from cartservice as shown in the screenshot below.

We can now go to Grafana cloud and add the cloud watch datasource with access credentials.

And should now be able to see the logs in Grafana for our log group, on the explore tab.

As a bonus let's try adding a panel that shows the no. of messages per severity.

Add a panel with query as shown in the screenshot below.

Go to the transform data tab and apply a transformation to parse the severity_text from the json message.

And then do a group by transformation to calculate the no. of messages per severity.

Can choose a visualization type such as pie chart like below.

The options I have set in the panel are as follows:

Title: Logs by severity
Value options > Show > All values
Legend > Legend values: Value, Percent

Ok that's it for this post, we saw how to get logs from the otel demo app via open collector to AWS CloudWatch Logs and explored it on Grafana with a sample visualization. We just need to delete the namespace with kubectl delete ns otelcol-logs-demo so that logs are not sent to AWS cloudwatch. Thanks for reading !!!