A Linux host is like a dude at a bar: after enough bad input it will panic, vomit all over its shoes and then try to carry-on like nothing had happened.
Let's use the
last tool to check wtmp login records and see if the last reboot was intentional or if the host automatically recovered from a crash.
Details of Linux user logins are stored in the (binary!) file
/var/log/wtmp and read with the
/bin/last utility. Output displays the user who logged-in, their source host or IP, and their login time and either their logout time or the status "still logged in":
VM: /bin/last lbonanomi pts/0 bastion_host Tue Sep 10 08:04 still logged in lbonanomi pts/0 bastion_host Tue Sep 10 07:39 - 07:43 (00:03)
Now by default
last only shows real users, but using the
-x argument will show logins for both real and pseudo users. Both
/sbinb/reboot write their names to wtmp records as part of their normal operation, so wtmp logs record an intentional reboot like this:
reboot system boot 3.10.0-229.el7.x Sat Sep 7 02:31 - 09:27 (3+06:56) shutdown system down 3.10.0-229.el7.x Sat Sep 7 02:31 - 02:31 (00:00)
With user "shutdown"'s login followed directly by user "reboot". An entry from user "reboot" not immediately followed by an entry from user "shutdown" should be considered suspicious enough to get an admin's attention.
Tuck this chunk of shell functionality into /etc/rc.local to check for crash-recovery and add hooks to disable any services that would cause contention:
( last -x -f $(ls -1t /var/log/wtmp* | head -2 | tail -1); last -x -f /var/log/wtmp ) | grep -A1 reboot | head -2 | grep -q shutdown || echo "panic-boot"