Introduction
In the world of modern web applications, monitoring and log management are crucial for maintaining system health and troubleshooting issues. This guide will walk you through the process of setting up the Datadog agent to collect Nginx logs from an AWS EC2 instance. We'll cover everything from initial installation to troubleshooting common issues.
Objectives
By the end of this guide, you will be able to:
- Install and configure the Datadog agent on an AWS EC2 instance
- Set up Nginx log collection
- Troubleshoot common issues in the setup process
- Verify successful log transmission to Datadog
Prerequisites
- An AWS EC2 instance running Ubuntu
- Nginx installed and running on the EC2 instance
- A Datadog account (sign up at https://www.datadoghq.com/)
Detailed Steps
1. Installing the Datadog Agent
First, we need to install the Datadog agent on our EC2 instance. We'll use the official installation script:
DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.eu" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
Replace <YOUR_API_KEY> with your actual Datadog API key. You can find this in your Datadog account settings.
Note: We're using datadoghq.eu here. If your Datadog organization is based in the US, use datadoghq.com instead.
2. Configuring the Datadog Agent
After installation, we need to configure the agent. The main configuration file is located at /etc/datadog-agent/datadog.yaml. Let's edit it:
sudo nano /etc/datadog-agent/datadog.yaml
In this file, we need to set up a few key parameters:
api_key: <YOUR_ACTUAL_API_KEY>
site: datadoghq.eu
cmd_port: 5002
expvar_port: 5002
logs_enabled: true
The cmd_port and expvar_port settings are important. We initially tried ports 5000 and 2001, but they were in use by other applications. Always check for available ports using:
sudo netstat -tulpn | grep LISTEN
3. Configuring Nginx Log Collection
Now, we need to tell Datadog which Nginx logs to collect. Create a new configuration file:
sudo nano /etc/datadog-agent/conf.d/nginx.d/conf.yaml
Add the following configuration:
logs:
- type: file
path: /var/log/nginx/access.log
service: nginx
source: nginx
- type: file
path: /var/log/nginx/error.log
service: nginx
source: nginx
This configuration tells Datadog to collect both access and error logs from Nginx.
datadog.yaml file code
sudo nano /etc/datadog-agent/datadog.yaml
api_key: YOUR_KEY
DD_API_KEY: 'YOUR_KEY'
DD_REMOTE_CONFIGURATION_ENABLED: false
remote_configuration_enabled: false
expvar_port: 5004
cmd_port: 5003
log_level: debug
log_to_console: true
forwarder_storage_max_size_in_bytes: 104857600
logs_enabled: true
logs_config:
container_collect_all: true
site: datadoghq.eu # .eu if your datadog is using Europe server else use .com
4. Setting Correct Permissions
One common issue is that the Datadog agent doesn't have permission to read the Nginx log files. Let's fix that:
sudo usermod -a -G adm dd-agent
sudo chmod 644 /var/log/nginx/*.log
sudo chmod 755 /var/log/nginx
These commands add the Datadog agent user to the adm group (which typically has access to log files) and ensure the log files and directory have the correct permissions.
5. Restarting and Verifying
After making these changes, restart the Datadog agent:
sudo systemctl restart datadog-agent
Now, let's verify our configuration:
sudo datadog-agent status
sudo datadog-agent configcheck
These commands will show the status of the agent and check for any configuration errors.
6. Monitoring Logs
To see what's happening in real-time, you can monitor the Datadog agent's logs:
sudo tail -f /var/log/datadog/agent.log
Look for any error messages or warnings here.
Troubleshooting Common Issues
API Key Issues
If you see a message like this:
WARN | api_key '***************************380ff' for domain https://api.datadoghq.com is invalid
Double-check your API key in the Datadog web interface and make sure it's correctly copied in your datadog.yaml file.
Permission Denied Errors
If you see:
WARN | open /var/log/nginx/error.log: permission denied
Revisit step 4 and ensure you've set the correct permissions.
Port Already in Use
If you encounter:
ERROR | Error while starting api server, exiting: unable to start CMD API server: unable to listen to the given address: listen tcp 127.0.0.1:5002: bind: address already in use
Choose a different port in your datadog.yaml file and restart the agent.
Verifying Success
Generate some traffic to your Nginx server, then check the Datadog web interface. Go to Logs > Search, and you should see your Nginx logs appearing.
Conclusion
Setting up the Datadog agent to collect Nginx logs from an AWS EC2 instance involves several steps and potential pitfalls. By following this guide, you should be able to successfully configure log collection and troubleshoot common issues. Remember, proper monitoring and log management are key to maintaining a healthy, performant web application.
Top comments (0)