Introduction
The other day it was the first time that I set up Datadog Agent in ECS on Fargate. I had a hard time understanding it even after reading the documentation, so I'm writing it down here.
Notes
- As the doc mentioned, if you wanna know how to set up EKS on Fargate, please be careful that the doc is different. Look at this document.
- Modify your container to the side-car container, not create another container for the setup.
- I'd misunderstood the first time that I had to create another container.
- By integrating the Agent, you can get some metrics from ECS Fargate.
- I'll eventually post about how to get the logs!
What the problem
First of all, the document only shows how to create it via WebUI, AWS CLI, or CloudFormation, which is confusing because I used Terraform when I created ECS.
I mean, I wanted to install Datadog Agent on a Fargate already built on Terraform, so I had no idea what to do at first.
But in this case, it is actually just a matter of modifying the task definition that already exists.
How to set up
Here is the my sample task definition, so I mean I only modify this file.
{
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "hoge-hoge-hoge-dev-cwlg-api-server",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "/ecs"
}
},
"portMappings": [
{
"protocol": "tcp",
"containerPort": 8080
}
],
"command": [
"sh",
"-c",
"npm run prisma:migrate:deploy && npm start"
],
"environment": [
{
"name": "SERVER_PORT_NUMBER",
"value": "8080"
},
{
"name": "AWS_S3_REGION",
"value": "ap-northeast-1"
},
{
"name": "TZ",
"value": "Asia/Tokyo"
},
{
"name": "SLACK_TOKEN",
"value": "hoge-99999969-60614-hoge-hoge-mEWS1wNZ0E765VkUvkM7"
}
],
"secrets": [
{
"name": "DATABASE_URL",
"valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_DATABASE_URL"
},
{
"name": "IMPORT_PROBLEM_END_POINT_SECRET",
"valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_END_POINT_SECRET"
},
{
"name": "COGNITO_USER_POOL_ID",
"valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_USER_POOL_ID"
}
],
"essential": true,
"pseudoTerminal": true,
"readonlyRootFilesystem": false,
"privileged": false,
"name": "app"
}
],
"memory": "2048",
"cpu": "1024",
"executionRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-exec",
"taskRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-task",
"family": "hoge-dev-task-api-server",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"]
}
It's a sample task def for Datadog agent.
{
"image": "datadog/agent:latest",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/hoge/hoge/datadog-agent-dev",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "ecs"
}
},
"cpu": 10,
"memory": 256,
"mountPoints": [],
"environment": [
{
"name": "ECS_FARGATE",
"value": "true"
},
{
"name": "DD_PROCESS_AGENT_ENABLED",
"value": "true"
},
{
"name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
"value": "true"
}
],
"secrets": [
{
"name": "DD_API_KEY",
"valueFrom": "/hoge-ssm/DATADOG_API_KEY"
}
],
"name": "api-server-datadog-agent"
}
What Params
| Key | Values | features | |
|---|---|---|---|
| image | datadog/agent:latest | Container Image | |
| environment: ECS_FARGATE | true | Set Agent ECS Fargate explicitly | |
| environment: DD_PROCESS_AGENT_ENABLED | true | Enabled live process collect | |
| environment: DD_DOGSTATSD_NON_LOCAL_TRAFFIC | true | Correct Metrics by DogStatsD on UDP8125 | |
| secrets: DD_API_KEY | /hoge-ssm/DATADOG_API_KEY | Your Datadog API KEY | |
| name | api-server-datadog-agent | My container name |
If you don't have the Datadog API key, you can get the organization page, let's please check the docs
Also, I put the small technique in that example, so I use SSM parameter store to avoid hard cording DD_API_KEY.
If you are interested in the tech, please look at the doc.
Finally, Completed the file💡
{
"containerDefinitions": [
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "hoge-hoge-hoge-dev-cwlg-api-server",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "/ecs"
}
},
"portMappings": [
{
"protocol": "tcp",
"containerPort": 8080
}
],
"command": [
"sh",
"-c",
"npm run prisma:migrate:deploy && npm start"
],
"environment": [
{
"name": "SERVER_PORT_NUMBER",
"value": "8080"
},
{
"name": "AWS_S3_REGION",
"value": "ap-northeast-1"
},
{
"name": "TZ",
"value": "Asia/Tokyo"
},
{
"name": "SLACK_TOKEN",
"value": "hoge-99999969-60614-hoge-hoge-mEWS1wNZ0E765VkUvkM7"
}
],
"secrets": [
{
"name": "DATABASE_URL",
"valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_DATABASE_URL"
},
{
"name": "IMPORT_PROBLEM_END_POINT_SECRET",
"valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_END_POINT_SECRET"
},
{
"name": "COGNITO_USER_POOL_ID",
"valueFrom": "/hoge-study-dev/api-secret-hoge/HOGE_USER_POOL_ID"
}
],
"essential": true,
"pseudoTerminal": true,
"readonlyRootFilesystem": false,
"privileged": false,
"name": "app"
},
,
{
"image": "datadog/agent:latest",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/hoge/hoge/datadog-agent-dev",
"awslogs-region": "ap-northeast-1",
"awslogs-stream-prefix": "ecs"
}
},
"cpu": 10,
"memory": 256,
"mountPoints": [],
"environment": [
{
"name": "ECS_FARGATE",
"value": "true"
},
{
"name": "DD_PROCESS_AGENT_ENABLED",
"value": "true"
},
{
"name": "DD_DOGSTATSD_NON_LOCAL_TRAFFIC",
"value": "true"
}
],
"secrets": [
{
"name": "DD_API_KEY",
"valueFrom": "/hoge-dev/api-secret-hoge/DATADOG_API_KEY"
}
],
"name": "api-server-datadog-agent"
}
],
"memory": "2048",
"cpu": "1024",
"executionRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-exec",
"taskRoleArn": "arn:aws:iam::99999999999:role/hoge-dev-role-api-server-task",
"family": "hoge-dev-task-api-server",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"]
}
Top comments (5)
Hello,
Sorry for a potentially silly question. But it is not mentioned on the rest of the doc and just in the final example. What is the ExecutionRoleArn, TaskRoleArn, and Family?
What do these fields do?
docs.aws.amazon.com/AmazonECS/late...
post the complete github link.
Hi, do you have a timeline when you'll post the log collection? I'm so confused by datadogs documentation for the integration.
Hello,
Do you have any other article to sending logs to datalog.
*By integrating the Agent, you can get some metrics from ECS Fargate.
I'll eventually post about how to get the logs!
*