The setup of datadog is so simple for anybody to do. But to get it to ship logs can be messy. This guide would help you in troubleshooting.
After installation and having all configurations right, this include
logs_enabled: true set in your
datadog.yaml, your datadog log setup may still not work.
These lessons are from experience and you can refer to this in times of need.
- Datadog Log
The first point of call is datadog logs. note: These logs are not managed by Logrotate so it can get very huge. I advise you
tail -n 500 /var/log/datadog/agent.log.
Check this log for TCP(Port) and Permission issues.
sudo cat /var/log/datadog/agent.log # or sudo cat /var/log/datadog/agent.log | grep permission
- Users and Usergroup
From experience datadog works well if the
/etc/datadog/conf.d/awesome-app.d/conf.yaml is owned by
dd-agent user and group you created during installation.
If you are not sure, run
cd /etc/datadog/conf.d/awesome-app && ll and view the usergrp and user. If the directory is owned by root or any other user, you can fix it with the following.
sudo chown dd-agent awesome-app.d/conf.yaml sudo chgrp dd-agent awesome-app.d/conf.yaml # and restart datadog sudo service datadog-agent restart
Verify if you are still unable to ship logs.
- Grant Permission to Datadog
Most of the time, we can get lucky by running some
chmod on the log file we are shipping to datadog. This is not so good since the directory and log is generated by an application with a different user and group.
So the best approach here is setfacl. It grants permission on a file without the need to compromise its previous users or permission.
Execute it as below per your path
setfacl -m u:dd-agent:rwx /var/log/awesome-app/file.log
or grant it on a directory
setfacl -m u:dd-agent:rwx /var/log/awesome-app
Restart and you would be fine.
Visit https://docs.datadoghq.com/ for more!