DEV Community

Cover image for Why Docker is eating your disk space
Juhana Jauhiainen
Juhana Jauhiainen

Posted on • Originally published at juhanajauhiainen.com

Why Docker is eating your disk space

I learned recently, in less than ideal circumstances, that Docker doesn't do any log rotation by default 😱

So if you're running your application in production with Docker and using Docker's default logging driver you might be awakened some night at 3 AM to fix a service outage when you can barely think straight. 😅

The default logging driver is json-file which caches the container logs as JSON. In practice, this means Docker is writing all the container stdout and stderr to a JSON file located in the host systems /var/lib/docker/containers folder.

You can find out the locations and sizes of these log files with docker inspect. You might need to run this with sudo.

du -h $(docker inspect --format='{{.LogPath}}' $(docker ps -qa))
Enter fullscreen mode Exit fullscreen mode

If you need to delete logs of a container before continuing, you can use echo to overwrite them as empty. This might require stopping the containers first.

echo "" > $(docker inspect --format='{{.LogPath}}' container_name)
Enter fullscreen mode Exit fullscreen mode

Changing the default logging driver

You can change the default logging driver by adding a configuration for it in /etc/docker/daemon.json. Docker supports multiple logging drivers but the simplest solution to our disk gorging logs problem is to use the local driver. The official Docker documentation explicitly refers to using it to prevent disk-exhaustion.

Let's create a basic config for local logging driver.

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  }
}
Enter fullscreen mode Exit fullscreen mode

We are defining the logging driver with log-driver and setting the maximum size of a log file before it is rolled. This means, when the file reaches 100 megabytes, a new file is created and the old one is archived. max-file is here set to "3", so at any point in time there will be only three log files stored. When the third file reaches 100 megabytes, a new file is created and the oldest log file is deleted.

Next, we need to reload the config for the Docker daemon. The Docker documentation suggests we can use SIGHUP signal to reload the configuration. Unfortunately, this doesn't work on all configuration options with logging driver being one of them.

This means we will have to restart the Docker daemon to reload the config. The command for restarting the daemon depends on the system it's running on. On Linux systems using [systemd] docker daemon can be restarted using the service command.

service docker restart
Enter fullscreen mode Exit fullscreen mode

Now if we check the logging driver currently in use with info, it should print local

docker info --format '{{.LoggingDriver}}'
Enter fullscreen mode Exit fullscreen mode

Now we have changed the default logging driver but if we have running containers this is not enough. The configuration change will only affect new containers. This means we will have to recreate all our existing containers. Depending on your setup you can do this using docker-compose or by running docker build

docker-compose up -d --force-recreate
Enter fullscreen mode Exit fullscreen mode
If you're using docker-compose and get an warning WARNING: no logs are available with the 'local' log driver when you're trying to read logs from a container, it means you're running an older version of docker-compose which doesn't support local logging driver. Try upgrading your docker-compose
Enter fullscreen mode Exit fullscreen mode




Further reading

Docker logging drivers
Configuring local driver

Photo by Fernando Lavin on Unsplash

Discussion (0)