Monitoring of AWS EKS using AWS Distro for OpenTelemetry (ADOT) and Amazon Managed Service for Prometheus (AMP)

In this tutorial, we will be using AWS Distro for OpenTelemetry to capture the metrics from AWS EKS and send them to Amazon managed service for Prometheus.

AWS Distro for OpenTelemetry(ADOT): - It is an AWS-supported version of the upstream OpenTelemetry Collector and is distributed by Amazon. It supports the selected components from the OpenTelemetry community. It is fully compatible with AWS computing platforms including EC2, ECS, and EKS. It enables users to send telemetry data to AWS CloudWatch Metrics, Traces, and Logs backends as well as the other supported backends.

Amazon Managed Service for Prometheus(AMP): - It is Prometheus-compatible monitoring and alerting service offered by AWS that makes it easy to monitor containerized applications and infrastructure at scale.

Prerequisite: -

1. Cert manager is installed and running. If it is not installed follow this URL to install it.
2. An AMP workspace is created. Guides for this can be found here.
3. If you are setting up the ADOT Collector of AWS EKS, you will need to set up IAM roles for service accounts for the ingestion of metrics from Amazon EKS clusters.

3.1. Open the IAM Console and edit the trust policy.
3.2. In the left navigation pane, choose Roles and find the amp-iamproxy-ingest-role that you created in Step 3.
3.3. Choose the Trust Relationships tab and choose Edit trust relationship.
3.4. In the trust relationship policy JSON, replace aws-amp with adot-col and choose Update Trust Policy. Your resulting trust policy should look like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::account-id:oidc-provider/oidc.eks.aws_region.amazonaws.com/id/openid"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.aws_region.amazonaws.com/id/openid:sub": "system:serviceaccount:adot-col:amp-iamproxy-ingest-service-account"
        }
      }
    }
  ]
}

3.5. Choose the Permissions tab and make sure that the following permissions policy is attached to the role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "aps:RemoteWrite",
                "aps:GetSeries",
                "aps:GetLabels",
                "aps:GetMetricMetadata"
            ],
            "Resource": "*"
        }
    ]
}

ADOT Installation

Assuming that you have all the prerequisites installed or created and you are ready to deploy the ADOT on your cluster.

• Before installing ADOT, we need to make sure that ADOT is configured in a way that it can send its metrics data to Amazon Managed Prometheus(AMP) and for this purpose, you have to first download the Prometheus file by running the following command.

wget https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/examples/eks/aws-prometheus/prometheus-sample-app.yaml

Once you downloaded this file. Now you need to change a few parameters in the file. These are mentioned below: -

• remote_write endpoint for your Amazon Managed Service for Prometheus workspace for YOUR_ENDPOINT and your Region for YOUR_REGION. You can get the remote_write URL from the AMP workspace which you have just created. Below is the screenshot of configmap, where you need to add your details.

In the above screenshot, we are using adot as the namespace. So when you check the metrics exported by your ADOT it will show your metrics name starting with “adot” Below is the screenshot which shows how your metrics will look.

So, for the metrics like this, you have to create your own custom dashboard on grafana. And if you want to use the already available dashboard then keep the namespace name empty like this “namespace: "”. Now it will not add any name in front of your metrics.

• You'll also need to change YOUR_ACCOUNT_ID in the service account section of the Kubernetes configuration to your AWS account ID.

• As the ADOT Prometheus Receiver supports the full set of Prometheus scraping and re-labelling configurations described in the Configuration section in the Prometheus documentation. You can paste these configurations directly into your ADOT Collector configurations. The configuration for the Prometheus Receiver includes your service discovery, scraping configurations, and re-labelling configurations. The receiver configurations look like the following.

receivers:
  prometheus:
    config:
      [Your Prometheus configuration]

You can download the file which we have used for this demo by using below mentioned command.
You can customise this file or you can use your own file.

wget https://raw.githubusercontent.com/sagar0419/Adot-Configuration/master/adot.yaml

Create a namespace in the kubernetes where you are goin to deploy the downloaded file.

 create ns adot-col

Once the namespace is created you can deploy your prometheus configuration.

 apply -f adot.yaml

You can verify the configuration once it is deployed on the cluster with the following command: -

  get all -n adot-col

If the configuration is deployed successfully, then you will get an output like this: -

Now your ADOT is deployed on the cluster. To check if it sending telemetric data to AMP or not, run the below command. But before running this command change the required parameters first. AMP_ENDPOINT, AMP_REGION.

(Note AWS AMP is a different AWS service which is not running on EKS so we cannot check it by running the kubectl commands).

awscurl --service="aps" --region="AMP_REGION" "https://AMP_ENDPOINT/api/v1/query?query=adot_test_gauge0"

You will get output similar to this.

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"name":"adot_test_gauge0"},"value":[1606512592.493,"16.87214000011479"]}]}}

Grafana for ADOT visualisation

Now we need to install Grafana on our Kubernetes cluster. You can deploy it anywhere or you can choose Grafana cloud also. For this tutorial we are going to deploy it on kubernetes.

To deploy grafana on your cluster you can use this helm chart. Please follow this document to enable sigv4 and IRSA in your grafana helm chart.

Once grafana is deployed login to the grafana console. Navigate to setting and select data source to add AWS Managed Prometheus as datasource.

Select Add data source, then Prometheus from the list as shown below:

Next, we paste the AMP Endpoint query URL (find this under the Summary tab on the AMP workspace) leaving out the api/v1/query portion (for example, https://aps-workspaces.us-west-2.amazonaws.com/workspaces/ws-3aa5f57b-yy11-xx00-12ab-ea86005d6dd7/) in the URL field under HTTP. We need to enable SigV4 auth in the Auth section.

We also need to ensure that AWS SDK Default is selected in Authentication Provider under the Sigv4 Auth Details section, then select the AWS Region in which the AMP workspace was created earlier in the Default Region drop-down. See the following screenshot for details.

Next, we can choose Save & Test. We should see a green banner that says “Data source is working” as shown in the following.

Query the metrics from AMP to verify the setup

Next, we’ll create a new Dashboard from the left navigation bar by choosing the + sign.

We then add a new panel and select the new AMP data source configured previously.

We can write a simple PromQL query in the Metrics textbox, and we should see the metrics in the panel as shown in the screenshot:

You can also create or upload a custom dashboard. In this demo we are using Node Exporter for Prometheus Dashboard. Below is the screenshot of the dashboard.

Now your cluster metrics are available on the grafana dashboard collected using ADOT and AMP.

Summary

In this blog post, we saw that how we can monitor our EKS cluster using AWS ADOT and AMP. We also saw that how we can export the metrics generated by EKS in Grafana.

I hope you found this post informative and engaging. I would love to hear your thoughts on this post, so do start a conversation on Twitter or LinkedIn .

Here are some of my other articles that you may find interesting.

Monitoring your application using OpenTelemetry and Jaeger.

Multitenancy in Kubernetes cluster using vCluster.

Backup and Restore Kubernetes Cluster.

Until Next time....