DEV Community

Serepas Filippas
Serepas Filippas

Posted on • Updated on

Docker Logs, set limits!

The problem...

I have a docker-compose.yaml which contains a simple python script. The job of this script is to subscribe to MQTT broker and then write these messages to a database.

The mistake...

Inside this script I have a simple innocent print statement, which prints the captured message.
We talk about 30 messages per second, and this service already runs for 3 plus months. So..., a lot of messages.

I need more space!!!

First of all, this script runs in Ubuntu server. At some point the hard disk doesn't have enough space. My first though is that there is much data inside my database.
But the problem here is that I get messages every second (~3600x30 messages!) which eventually I replace with their average value per hour. Again I think the problem is the database.

VACUUM IT!!!

I use Postgresql, so, I read the documentation.
Here we are! Let's release some space. At this point I don't want to use the VACUUM FULL because it causes downtimes...
After that, nothing changed dramatically in my server's hard drive..

Docker df

I think my first move should be this one. Check the space which is taken by my docker container,

docker system df
Enter fullscreen mode Exit fullscreen mode

Hmmm, here I am missing ~6.5Gb of space which at that point I assumed was being taken up by docker... somehow...

Logs, Logs everywhere..

Next move is to see the logs of the container

docker logs <container id>
Enter fullscreen mode Exit fullscreen mode

I see all these print statements which start from the beginning of container's existence.
So, I thought where these logs are stored?

 docker inspect --format='{{.LogPath}}' <my-app or container-id>
Enter fullscreen mode Exit fullscreen mode

The result will be something like this,

/var/lib/docker/containers/46d66d306b4a7f95051dbf2aa7b27864d78d84bfac7dc685c4b94985a1ee30c5/46d66d306b4a7f95051dbf2aa7b27864d78d84bfac7dc685c4b94985a1ee30c5-json.log
Enter fullscreen mode Exit fullscreen mode

Here you are!!!
Ok, then I cd to this directory and run,

ls -l -h
Enter fullscreen mode Exit fullscreen mode

To see 6.5Gb of logs!!!
Here was my precious Gb!!!

How clear this logs?

The first fast and easy way, without need to re-build your container is,

sudo sh -c 'echo "" > $(docker inspect --format="{{.LogPath}}" my-app)'
Enter fullscreen mode Exit fullscreen mode

Be aware, this is a temporary solution! You will come up with the same problem in the near future!

Set limits in docker-compose

You can set limits of logs for your container in your docker-compose.yaml

logging:
      driver: "json-file"
      options:
        max-size: 50m
Enter fullscreen mode Exit fullscreen mode

Here we set limit of 50mb of log file.

Set limits in docker run

docker run \
      --log-driver local --log-opt max-size=10m \
      alpine echo hello world
Enter fullscreen mode Exit fullscreen mode

Another example, set 10mb limit of logs with docker command
To be continued...

Sources

Oldest comments (0)