DEV Community

Leon Nunes
Leon Nunes

Posted on • Updated on

How a simple Hang-up borked my lab environment.

Happy New Year! Well my new year hasn't been that happy, from falling sick on the first day of the year to my homelab giving up, life has just been going on...

Upgrading never seemed risky until one day my Arch homelab(Yes I run Arch :o) hung up and caused all the kernels to get deleted. started getting funky GPG and corrupted package errors.

It all began here...

Linux failing to find the kernel

So I popped in a usb and started a live environment, mounted my folders and did arch-chroot, then tried to do a pacman -Syu only to get hit with a bunch of:

error: GPGME error: No data
Enter fullscreen mode Exit fullscreen mode

So I thought let me just simply fix it by following the Arch guide, however, nothing worked, what did was

sudo rm -fr /etc/pacman.d/gnupg
sudo pacman-key --init
sudo pacman-key --populate archlinux manjaro
sudo pacman-key --refresh-keys
Enter fullscreen mode Exit fullscreen mode

I also got few errors of files being already owned, since I was sleepy I did some bash-fu and ended up with no /usr :D, fret not I copied it from my live system(Spoiler Alert: It borked up systemD). It worked until it didn't.

Anna Kendrick doing Boom sign

So then I figured out somehow that SystemD and QEMU was failing in some ways with Permission Denied errors and Qemu monitor getting a similar error, a quick reinstall fixed it...

Weirdly this was a chain reaction... I booted up my PC and I kept getting an error that said I've overclocked it... but I never did that, this caused the bios to eventually wipe everything on every boot. Which means I couldn't run VMs because the Virtualization option never persisted.

So I did the most sensible thing I replaced the CMOS battery, cause this is a really old system. Even then it still didn't work. Then I noticed that the Ram was showing up as 8GB, this system had 16GB ram and so it turned out that a faulty Memory stick brought my system down for 2 good days...

Surprisingly, DDR3 is quite cheap so I'll be replacing that Ram Stick.

That's it folks, I also got a new box to play around these days and I'm wondering what OS should I put on it any suggestions!?

Top comments (2)

Collapse
 
delusionaloptimist profile image
Rudraksh Pareek

Interesting read! You should maybe give NixOS a try. Haven't used it myself but a friend won't stop talking about how convenient it is to have an entirely "declarative" system :P

Collapse
 
mediocredevops profile image
Leon Nunes

Honestly its on my list, I tried it once and then left it, I need a new OS for my homelab might use that, the kernel disappearing was a bit scary I wonder how that happened.