DEV Community

Cover image for How I messed up my systemd logs in NixOS and how I fixed it.
Lucas Eduardo
Lucas Eduardo

Posted on

How I messed up my systemd logs in NixOS and how I fixed it.

NixOS is an amazing distro, like, imagine configuring all your system stuff, containers, services, programs, development environments and so on using a consistent language. This is not the future but actually the present.

Even though NixOS modules can do validation of inputs and scenarios such as not allowing you to have both pipewire and pulseaudio in the same system, because it's known that it will conflict with each other, there are stuff that modules can't predict, and at the end of the day modules are made by humans and humans do make mistakes even though there is a lot of code review happening on nixpkgs.

This case is very unlikely to be treated by a module because it's more like a runtime issue.

Systemd-journald grants access to users to view their service logs using ACLs, that are basically an extension to what chmod does. Root and the systemd-logs have write access to the logs and the final user only has the privilege to read them using a special annotation. The problem is that my zfs filesystem wasn't allowing this annotation to be saved because zfs doesn't support saving these ACLs by default. For it to save you have to set the acltype attribute to posixacl using zfs set. I did know about this when I built the partition scheme for this system but I missed a small little thing: It was mounting that dataset on the wrong location: /var/journal instead of /var/log/journal.

To fix this, first I stopped the journal using systemctl stop systemd-journald so it would not try to write to that location. Second, I moved /var/log/journal to /var/log/journal.old then created an empty folder on that location so I can remount that dataset that was mounted on the wrong place to /var/log/journal. After that I copied using rsync from /var/log/journal.old to /var/log/journal, fixed the ACLs using sudo setfacl u:lucasew:r 97e3b5a7928145bdb0d45bc645b5c20e/user-1000* (you will have to change it to your own user and user id) and then tested running journalctl --user -r. Stuff then appeared nicely so it's fixed. Finally I ran systemctl start systemd-journald to start the journal service again and changed in my hardware-configuration.nix so the next time it will mount that dataset correctly.

Top comments (0)