Setting Up Datadog Agent for Nginx Log Collection on AWS EC2

Introduction

In the world of modern web applications, monitoring and log management are crucial for maintaining system health and troubleshooting issues. This guide will walk you through the process of setting up the Datadog agent to collect Nginx logs from an AWS EC2 instance. We'll cover everything from initial installation to troubleshooting common issues.

Objectives

By the end of this guide, you will be able to:

1. Install and configure the Datadog agent on an AWS EC2 instance
2. Set up Nginx log collection
3. Troubleshoot common issues in the setup process
4. Verify successful log transmission to Datadog

Prerequisites

• An AWS EC2 instance running Ubuntu
• Nginx installed and running on the EC2 instance
• A Datadog account (sign up at https://www.datadoghq.com/)

Detailed Steps

1. Installing the Datadog Agent

First, we need to install the Datadog agent on our EC2 instance. We'll use the official installation script:

DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.eu" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

Replace <YOUR_API_KEY> with your actual Datadog API key. You can find this in your Datadog account settings.

Note: We're using datadoghq.eu here. If your Datadog organization is based in the US, use datadoghq.com instead.

2. Configuring the Datadog Agent

After installation, we need to configure the agent. The main configuration file is located at /etc/datadog-agent/datadog.yaml. Let's edit it:

sudo nano /etc/datadog-agent/datadog.yaml

In this file, we need to set up a few key parameters:

api_key: <YOUR_ACTUAL_API_KEY>
site: datadoghq.eu
cmd_port: 5002
expvar_port: 5002
logs_enabled: true

The cmd_port and expvar_port settings are important. We initially tried ports 5000 and 2001, but they were in use by other applications. Always check for available ports using:

sudo netstat -tulpn | grep LISTEN

3. Configuring Nginx Log Collection

Now, we need to tell Datadog which Nginx logs to collect. Create a new configuration file:

sudo nano /etc/datadog-agent/conf.d/nginx.d/conf.yaml

Add the following configuration:

logs:
  - type: file
    path: /var/log/nginx/access.log
    service: nginx
    source: nginx
  - type: file
    path: /var/log/nginx/error.log
    service: nginx
    source: nginx

This configuration tells Datadog to collect both access and error logs from Nginx.

datadog.yaml file code

sudo nano /etc/datadog-agent/datadog.yaml

api_key: YOUR_KEY
DD_API_KEY: 'YOUR_KEY'
DD_REMOTE_CONFIGURATION_ENABLED: false
remote_configuration_enabled: false
expvar_port: 5004
cmd_port: 5003
log_level: debug
log_to_console: true
forwarder_storage_max_size_in_bytes: 104857600
logs_enabled: true
logs_config:
  container_collect_all: true

site: datadoghq.eu # .eu if your datadog is using Europe server else use .com

4. Setting Correct Permissions

One common issue is that the Datadog agent doesn't have permission to read the Nginx log files. Let's fix that:

sudo usermod -a -G adm dd-agent
sudo chmod 644 /var/log/nginx/*.log
sudo chmod 755 /var/log/nginx

These commands add the Datadog agent user to the adm group (which typically has access to log files) and ensure the log files and directory have the correct permissions.

5. Restarting and Verifying

After making these changes, restart the Datadog agent:

sudo systemctl restart datadog-agent

Now, let's verify our configuration:

sudo datadog-agent status
sudo datadog-agent configcheck

These commands will show the status of the agent and check for any configuration errors.

6. Monitoring Logs

To see what's happening in real-time, you can monitor the Datadog agent's logs:

sudo tail -f /var/log/datadog/agent.log

Look for any error messages or warnings here.

Troubleshooting Common Issues

API Key Issues

If you see a message like this:

WARN | api_key '***************************380ff' for domain https://api.datadoghq.com is invalid

Double-check your API key in the Datadog web interface and make sure it's correctly copied in your datadog.yaml file.

Permission Denied Errors

If you see:

WARN | open /var/log/nginx/error.log: permission denied

Revisit step 4 and ensure you've set the correct permissions.

Port Already in Use

If you encounter:

ERROR | Error while starting api server, exiting: unable to start CMD API server: unable to listen to the given address: listen tcp 127.0.0.1:5002: bind: address already in use

Choose a different port in your datadog.yaml file and restart the agent.

Verifying Success

Generate some traffic to your Nginx server, then check the Datadog web interface. Go to Logs > Search, and you should see your Nginx logs appearing.

Conclusion

Setting up the Datadog agent to collect Nginx logs from an AWS EC2 instance involves several steps and potential pitfalls. By following this guide, you should be able to successfully configure log collection and troubleshoot common issues. Remember, proper monitoring and log management are key to maintaining a healthy, performant web application.