Linux: persistent logs in systemd-journald

setevoy profile image Arseny Zinchenko Originally published at rtfm.co.ua on ・2 min read

We have an AWS EC2 which suddenly rebooted.

The issue is that its /var/log/messages incomplete and the latest record is about rsyslog service killed – but nothing about who killed it and why.

Then I went to the journald but "surprise" – it stores logs only for the current boot so there was no way to find the reboot’s root cause.:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
0 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:48:44 EEST

So, we need to configure journald to store logs after each reboot.

Its config file /etc/systemd/journald.conf.

To make logs to be stored always – update the Storage parameter to the auto or persistent.

If auto – then the /var/run/journal directory must be created manually, with the persistentsystemd will create it.

Update config, set persistent, restart systemd-journald to apply:

root@bttrm-stage-console:/home/admin# systemctl restart systemd-journald

Check the logs directory:

root@bttrm-stage-console:/home/admin# ll /var/log/journal/
total 4
drwxr-xr-x 2 root root 4096 Sep 24 17:54 7066493f616c4a6285c19cfaa2884b09

root@bttrm-stage-console:/home/admin# ll /var/log/journal/7066493f616c4a6285c19cfaa2884b09/
total 57348
-rw-r----- 1 root root 58720256 Sep 24 17:54 system.journal

Reboot the EC2 and check logs again:

root@bttrm-stage-console:/home/admin# journalctl --list-boots
-1 f527010076a141c5917496c6aa03438a Tue 2019-09-24 12:13:02 EEST—Tue 2019-09-24 17:56:19 EEST

0 591136b94276490580f4e39e576e639e Tue 2019-09-24 17:56:20 EEST—Tue 2019-09-24 17:56:41 EEST

Now you are able to check the previous reboot log:

root@bttrm-stage-console:/home/admin# journalctl -b -1

-- Logs begin at Tue 2019-09-24 12:13:02 EEST, end at Tue 2019-09-24 17:57:36 EEST. --

Sep 24 12:13:02 bttrm-stage-console CRON[19407]: pam\_unix(cron:session): session closed for user me-v3

Size can be limited by setting the SystemMaxUse, see the documentation.


Similar posts

Posted on Oct 7 '19 by:

setevoy profile

Arseny Zinchenko


DevOps, cloud and infrastructure engineer. Love Linux, OpenSource, and AWS.


markdown guide