If you are lucky, the disk usage was already monitored and you have been warned on time. Otherwise it's already too late and some of your apps are simply not working anymore. To help you get back on track asap, here are some useful commands I found throughout the years to quickly and safely made room again on your instance / server.
First connect to it via SSH of course and become root with sudo su -
Then given your case, if you are using on it Docker to run containers :
- if your app is still running, clean Docker resources smoothly
docker images -q | xargs docker rmi
docker container prune -f
docker image prune -a -f
docker system prune -a -f
docker volume prune -f
- if you app is already down, clean all Docker resources in one shot
docker system prune
- if a container does not rotate its log, that may represent a huge file, go clean it
cd /var/lib/docker/containers/<container_id>/ # a file <container_id>-json.log is supposed to be there
truncate -s 0 <container_id>-json.log
If you are using a Linux distribution with systemd, you can clean its logs:
cd /var/log/journal # they are here
journalctl --disk-usage # have a look at how much you can reclaim
journalctl --vacuum-time=7d # reclaim it (7d = keep logs from last week only)
Look elsewhere on the instance if you still need more space :
du -sh * | grep "G" # travel around to find most imposant directories
- if it's a log file you can truncate it :
truncate -s 0 YOUR_LOGFILE.log
- if you find a directory containing old files that are not needed anymore:
pwd # DOUBLE CHECK that you are inside the right directory to clean
find * -type f -mtime +100 -exec rm -rf {} \; # remove files not modified since 100 days for example
You should be good now and your apps should be back and running again, maybe you will need to redeploy / restart them for that.
Finally it's time now for continuous improvement :
- make sure the disk space on this instance is well monitored and the threshold in place does leave you enough time to intervene
- your goal is to never do that again : find easy cleaning commands that you can put in the crontab of this instance for example
Please let us know if this article helped you in some ways, I would love to hear your stories regarding this usecase (never funny) and your personal tips and suggestions too.
I wish you a great day!
Top comments (0)