I log, you log, we log for the love of logs

The number of Docker containers deployed in our production boxes have steadily grown over the past 3 years. Last count - 284 containers running. I love logging but I'm shy to say we only just implemented centralised logging for these Docker containers yesterday.

We are a profitable bootstrapped startup, and we don't have VC's money bags 💰 and unicorns 🦄 to chase. At TAGGUN, our top priority is to serve our customers by building receipt and invoice OCR scanning API that doesn't suck. So, the power of scarcity has deemed centralised logging as a YAGNI until a few days ago.

I wished I had a cheatsheet to implement centralised logging for Docker containers. So, I'm sharing it here with you to save you time and money.

How is it done?

• Jump to the Docker host
• Create or Modify the daemon.json found at /etc/docker
• You may need to sudo -i first to access this area as the root user.
• Restart the docker daemon (this will take this box down for about 15-20 seconds) sudo systemctl restart docker
• Verify the docker logger docker info --format '{{.LoggingDriver}}'
• docker ps (This is to validate that docker has restarted without issues)

What configuration for daemon.json?

{
    "log-driver": "syslog",
    "log-opts": {
        "labels": "docker-<machine_name>",
        "tag": "docker-<machine_name>[{{.ImageName}}/{{.Name}}]",
        "mode": "non-blocking"
    }
}

Breakdown

• log-driver is built in logging driver for docker; syslog is the one we're using because many server monitoring agents can easily read them from Syslogs.
• labels don't get logged to syslog it seems; left in because of docker best practices.
• tag does get logged to syslog. This is how you can let your log viewers knows what the application name, and processes are.
• mode is set to non-blocking which means a memory buffer is used for logging to syslog. if the log backpressure exceeds a certain point, logs will simply be dropped from this buffer, rather than blocking.

My recommended logging as a service platforms

• Site24x7
• Loggly
• Sematext
• Elastic Cloud (I love Elasticsearch and ELK stack. But my managed cluster is down too frequently)

[Cover photo by Andrew Ridley on Unsplash]