Did Linux Panic?

A Linux host is like a dude at a bar: after enough bad input it will panic, vomit all over its shoes and then try to carry-on like nothing had happened.

Let's use the last tool to check wtmp login records and see if the last reboot was intentional or if the host automatically recovered from a crash.

Details of Linux user logins are stored in the (binary!) file /var/log/wtmp and read with the /bin/last utility. Output displays the user who logged-in, their source host or IP, and their login time and either their logout time or the status "still logged in":

VM: /bin/last
lbonanomi pts/0        bastion_host Tue Sep 10 08:04   still logged in
lbonanomi pts/0        bastion_host Tue Sep 10 07:39 - 07:43  (00:03)

Now by default last only shows real users, but using the -x argument will show logins for both real and pseudo users. Both /sbin/shutdown and /sbinb/reboot write their names to wtmp records as part of their normal operation, so wtmp logs record an intentional reboot like this:

reboot   system boot  3.10.0-229.el7.x Sat Sep  7 02:31 - 09:27 (3+06:56)
shutdown system down  3.10.0-229.el7.x Sat Sep  7 02:31 - 02:31  (00:00)

With user "shutdown"'s login followed directly by user "reboot". An entry from user "reboot" not immediately followed by an entry from user "shutdown" should be considered suspicious enough to get an admin's attention.

Tuck this chunk of shell functionality into /etc/rc.local to check for crash-recovery and add hooks to disable any services that would cause contention:

(
  last -x -f $(ls -1t /var/log/wtmp* | head -2 | tail -1);  
  last -x -f /var/log/wtmp
) | grep -A1 reboot | head -2 | grep -q shutdown || echo "panic-boot"