As my first post here on dev.to, I have decided to share my little note on how to quickly setup up an environment for linux kernel module debugging in QEMU.
Download and extract linux kernel and busybox sources.
$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.tar.xz $ tar xvf linux-5.6.tar.xz $ wget https://busybox.net/downloads/busybox-1.31.1.tar.bz2 $ tar xvf busybox-1.31.1.tar.bz2
Configure and build linux kernel as usual. A couple of points:
- You can safely start with
defconfig, enabling options your module depends on.
- Make sure that all options you will need during debugging are compiled-in and not built as modules, so you don't have to cram them into the initramfs.
- Enable at least these options if you want to use gdb (see "Kernel hacking" section in
CONFIG_DEBUG_KERNEL=y(enables kernel debugging facilities)
CONFIG_DEBUG_INFO=y(includes debug symbols)
CONFIG_KGDB=y(enables kernel GDB backend over serial line)
CONFIG_PROVE_LOCKING=y(enable lock dependency checker; not needed for gdb to work, but comes handy for deadlock debugging).
You can quickly check if your kernel boots at all:
qemu-system-x86_64 -kernel linux-5.6.2/arch/x86_64/boot/bzImage \ -nographic \ -append "console=ttyS0"
Now, it is time to build our busybox userland. Again,
defconfig will do just fine, with a couple of points:
- Busybox must be linked statically: set
CONFIG_STATIC=y(found in "Settings" - "Build Options" section)
- Expect surprises when building busybox on a system with latest glibc versions. In the case of Archlinux, I've got the following:
netstat.c:(.text.ip_port_str+0x50): warning: Using 'getservbyport' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking ... rdate.c:(.text.rdate_main+0xf8): undefined reference to `stime'
You can get around both problems by building busybox with musl instead if glibc:
# Archlinux sudo pacman -S musl kernel-headers-musl make CC=musl-gcc # Ubuntu sudo apt install musl musl-dev musl-tools CFLAGS="-I../linux-5.6/usr/include" make CC=musl-gcc
If using kernel headers from the kernel tarball, you have to install them first locally (
Alternatively, you can try to proceed with glibc by applying this patch to fix stime calls (of course, this wont fix glibc broken static linking warnings).
Busybox binary and ab init script are pretty much all we need for our simple initramfs. Let's put busybox in to /bin and create a /bin/sh symlink, so kernel can look up the shell for our init script:
mkdir bin cp ../busybox-1.31.1/busybox bin/ ln -s /bin/busybox bin/sh touch init && chmod 777 init
Our init script should at least do the following:
- mount procfs and sysfs;
- populate /dev using information provided by kernel at sysfs (busybox's mdev can nicely do it for us);
- do your initialization (load a module for debugging, run a test program, you name it); the snippet below also contains an example on how to run some code on per-VM instance basis, which is needed when you are debugging some module that has to communicate to its remote counterpart on another machine;
- drop to interactive shell.
Here's the init script template I use:
#!/bin/sh # Install busybox applets as symlinks /bin/busybox --install -s /bin # Mount procfs & sysfs, populate /dev mkdir proc sys mount -t proc none /proc mount -t sysfs none /sys mdev -s # Optional dynamic device creation: # 1. Old method using mdev as an uevent helper # echo /bin/mdev > /proc/sys/kernel/hotplug # 2. New method using Netlink (requires busybox 1.31 or newer) # mdev -d # Get virtual machine instance number passed as a kernel argument inst=$(cat /proc/cmdline | grep -Eo 'vm_inst=[^ ]+' | cut -d '=' -f 2) # Do your common initialization here (insert debugged module, run some test script, etc) echo the init is running # Do your per-vm instance initialization here (assign an IP address, etc) [ $inst == 1 ] && echo first instance init [ $inst == 2 ] && echo second instance init # Drop to the shell exec setsid cttyhack sh
A couple of notes:
- Automatic device creation will obviously not work with this simplistic setup, so when, say, you create a character device in your driver, you have to start
mdev -sagain manually.
- In order to enable hotplugging, you have to either use busybox 1.31 or later, witch introduced netlink interface support for mdev (
mdev -doption). Alternatively, you can enable legacy
CONFIG_UEVENT_HELPERoption in Linux (disabled by default) and set hotplug executable to mdev:
echo /bin/mdev > /proc/sys/kernel/hotplug.
- It is not very convenient to just drop to shell using
exec /bin/shbecause at startup the
/dev/consoleis used by default as a tty. The problem is
/dev/consolecan't be a controlling terminal, thus rendering the job control stuff broken (usually indicated by
can't access tty; job control turned offmessage). This can be easily worked around with
setsid cttyhack sh.
setsidwill exec a cttyhack as a session leader, which will enable job control, and
cttyhackwill find a real tty device (usually ttyS0) and reopen stdin/stdout.
Finally, let's generate the initramfs image (assuming
initramfs is your initramfs root containing
/init script and the
cd initramfs find . -print0 | cpio --null -ov -H newc | gzip -9 > ../initramfs.tgz
Running QEMU is as easy as:
qemu-system-x86_64 \ -enable-kvm -cpu host -smp 1 \ -m 256M \ -nographic \ -kernel linux-5.6.2/arch/x86_64/boot/bzImage \ -initrd initramfs.tgz \ -chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 \ -device pci-serial,chardev=gdb \ -append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS1,115200"
-enable-kvm -cpu host -smp 2enables KVM acceleration and sets emulated CPU model to the host CPU;
-smpsets the number of CPU cores the guest can use (1 by default);
-m 256Mspecifies the amount of memory guest can use (128M by default);
-nographicdisables graphical output and instead emulates a serial port (
ttyS0) attached to stdout/stdin;
-initrdare quite self explanatory;
-chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0creates a socket listening for GDB frontend connections on TCP port 1234; this socket is later bound to the virtual serial port, which will appear as
ttyS1in the VM;
-device pci-serial,chardev=gdbcreates an emulated PCI serial port device, redirecting its IO to the TCP socket;
- -append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS1,115200" specifies the kernel command line; remove
kgdbwaitto let the kernel boot without waiting for GDB connection.
Attaching the gdb to the running kernel is also easy:
$ cd linux-5.6 $ gdb vmlinux (gdb) target remote :1234 (gdb) c
You'll probably need to explicitly allow gdb to load linux kernel debugging helper scripts. GDB will nicely warn about this if auto-loading is declined:
echo 'add-auto-load-safe-path /home/dav/Dev/qemu/linux-5.6/scripts/gdb/vmlinux-gdb.py' >> ~/.gdbinit
A bit more evolved example: running two machine instances communicating over emulated null modem.
qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -smp 2 \ -nographic \ -kernel linux-5.6.2/arch/x86_64/boot/bzImage \ -initrd initram.tgz \ -chardev socket,id=comm,port=8909,host=127.0.0.1 \ -device pci-serial,chardev=comm \ -chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 \ -device pci-serial,chardev=gdb \ -append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS2,115200"
qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -smp 2 \ -nographic \ -kernel linux-5.6.2/arch/x86_64/boot/bzImage \ -initrd initram.tgz \ -chardev socket,port=8909,server,id=comm,host=0.0.0.0 \ -device pci-serial,chardev=comm \ -chardev socket,id=gdb,port=1235,server,nowait,host=0.0.0.0 \ -device pci-serial,chardev=gdb \ -append "console=ttyS0 vm_inst=2 kgdbwait kgdboc=ttyS2,115200"
And that's all for now :) Hope you'll find this useful.