DEV Community

Cover image for How to integrate Datadog Agent in ECS Fargate
Masayoshi Haruta for AWS Community Builders

Posted on • Updated on

 

How to integrate Datadog Agent in ECS Fargate

Introduction

The other day it was the first time that I set up Datadog Agent in ECS on Fargate. I had a hard time understanding it even after reading the documentation, so I'm writing it down here.

Notes

  • As the doc mentioned, if you wanna know how to set up EKS on Fargate, please be careful that the doc is different. Look at this document.
  • Modify your container to the side-car container, not create another container for the setup.
    • I'd misunderstood the first time that I had to create another container.
  • By integrating the Agent, you can get some metrics from ECS Fargate.
    • I'll eventually post about how to get the logs!

What the problem

First of all, the document only shows how to create it via WebUI, AWS CLI, or CloudFormation, which is confusing because I used Terraform when I created ECS.

I mean, I wanted to install Datadog Agent on a Fargate already built on Terraform, so I had no idea what to do at first.
But in this case, it is actually just a matter of modifying the task definition that already exists.

How to set up

Here is the my sample task definition, so I mean I only modify this file.

{
    "containerDefinitions": [
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "hoge-hoge-hoge-dev-cwlg-api-server",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "/ecs"
                }
            },
            "portMappings": [
                {
                    "protocol": "tcp",
                    "containerPort": 8080
                }
            ],
            "command": [
                "sh",
                "-c",
                "npm run prisma:migrate:deploy && npm start"
            ],
            "environment": [
                {
                    "name": "SERVER_PORT_NUMBER",
                    "value": "8080"
                },
                {
                    "name": "AWS_S3_REGION",
                    "value": "ap-northeast-1"
                },
                {
                    "name": "TZ",
                    "value": "Asia/Tokyo"
                },
                {
                    "name": "SLACK_TOKEN",
                    "value": "hoge-99999969-60614-hoge-hoge-mEWS1wNZ0E765VkUvkM7"
                }
            ],
            "secrets": [
                {
                    "name": "DATABASE_URL",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_DATABASE_URL"
                },
                {
                    "name": "IMPORT_PROBLEM_END_POINT_SECRET",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_END_POINT_SECRET"
                },
                {
                    "name": "COGNITO_USER_POOL_ID",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_USER_POOL_ID"
                }
            ],
            "essential": true,
            "pseudoTerminal": true,
            "readonlyRootFilesystem": false,
            "privileged": false,
            "name": "app"
        }
    ],
    "memory": "2048",
    "cpu": "1024",
    "executionRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-exec",
    "taskRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-task",
    "family": "hoge-dev-task-api-server",
    "networkMode": "awsvpc",
    "requiresCompatibilities": ["FARGATE"]
}
Enter fullscreen mode Exit fullscreen mode

It's a sample task def for Datadog agent.

{
    "image": "datadog/agent:latest",
    "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "/hoge/hoge/datadog-agent-dev",
            "awslogs-region": "ap-northeast-1",
            "awslogs-stream-prefix": "ecs"
        }
    },
    "cpu": 10,
    "memory": 256,
    "mountPoints": [],
    "environment": [
        {
            "name": "ECS_FARGATE",
            "value": "true"
        },
        {
            "name": "DD_PROCESS_AGENT_ENABLED",
            "value": "true"
        },
        {
            "name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
            "value": "true"
        }
    ],
    "secrets": [
        {
            "name": "DD_API_KEY",
            "valueFrom": "/hoge-ssm/DATADOG_API_KEY"
        }
    ],
    "name": "api-server-datadog-agent"
}
Enter fullscreen mode Exit fullscreen mode

What Params

Key Values features
image datadog/agent:latest Container Image
environment: ECS_FARGATE true Set Agent ECS Fargate explicitly
environment: DD_PROCESS_AGENT_ENABLED true Enabled live process collect
environment: DD_DOGSTATSD_NON_LOCAL_TRAFFIC true Correct Metrics by DogStatsD on UDP8125
secrets: DD_API_KEY /hoge-ssm/DATADOG_API_KEY Your Datadog API KEY
name api-server-datadog-agent My container name

If you don't have the Datadog API key, you can get the organization page, let's please check the docs
Datadog API key

Also, I put the small technique in that example, so I use SSM parameter store to avoid hard cording DD_API_KEY.
If you are interested in the tech, please look at the doc.
SSM param

Finally, Completed the file💡

{
    "containerDefinitions": [
        {
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "hoge-hoge-hoge-dev-cwlg-api-server",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "/ecs"
                }
            },
            "portMappings": [
                {
                    "protocol": "tcp",
                    "containerPort": 8080
                }
            ],
            "command": [
                "sh",
                "-c",
                "npm run prisma:migrate:deploy && npm start"
            ],
            "environment": [
                {
                    "name": "SERVER_PORT_NUMBER",
                    "value": "8080"
                },
                {
                    "name": "AWS_S3_REGION",
                    "value": "ap-northeast-1"
                },
                {
                    "name": "TZ",
                    "value": "Asia/Tokyo"
                },
                {
                    "name": "SLACK_TOKEN",
                    "value": "hoge-99999969-60614-hoge-hoge-mEWS1wNZ0E765VkUvkM7"
                }
            ],
            "secrets": [
                {
                    "name": "DATABASE_URL",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_DATABASE_URL"
                },
                {
                    "name": "IMPORT_PROBLEM_END_POINT_SECRET",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_END_POINT_SECRET"
                },
                {
                    "name": "COGNITO_USER_POOL_ID",
                    "valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_USER_POOL_ID"
                }
            ],
            "essential": true,
            "pseudoTerminal": true,
            "readonlyRootFilesystem": false,
            "privileged": false,
            "name": "app"
        },
        ,
        {
            "image": "datadog/agent:latest",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/hoge/hoge/datadog-agent-dev",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "cpu": 10,
            "memory": 256,
            "mountPoints": [],
            "environment": [
                {
                    "name": "ECS_FARGATE",
                    "value": "true"
                },
                {
                    "name": "DD_PROCESS_AGENT_ENABLED",
                    "value": "true"
                },
                {
                    "name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
                    "value": "true"
                }
            ],
            "secrets": [
                {
                    "name": "DD_API_KEY",
                    "valueFrom": "/hoge-dev/api-secret-hoge/DATADOG_API_KEY"
                }
            ],
            "name": "api-server-datadog-agent"
        }
    ],
    "memory": "2048",
    "cpu": "1024",
    "executionRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-exec",
    "taskRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-task",
    "family": "hoge-dev-task-api-server",
    "networkMode": "awsvpc",
    "requiresCompatibilities": ["FARGATE"]
}
Enter fullscreen mode Exit fullscreen mode

Top comments (3)

Collapse
 
nialljude profile image
Niall

Hello,

Sorry for a potentially silly question. But it is not mentioned on the rest of the doc and just in the final example. What is the ExecutionRoleArn, TaskRoleArn, and Family?

What do these fields do?

Collapse
 
patrickcaylent profile image
patrick-caylent
Collapse
 
neomrc profile image
Marc

Hi, do you have a timeline when you'll post the log collection? I'm so confused by datadogs documentation for the integration.

Timeless DEV post...

Git Concepts I Wish I Knew Years Ago

The most used technology by developers is not Javascript.

It's not Python or HTML.

It hardly even gets mentioned in interviews or listed as a pre-requisite for jobs.

I'm talking about Git and version control of course.

One does not simply learn git