How to set up binfmt_misc for qemu the hard way

I recently discussed about options to run x86 binaries on a pantavisor powered ARM system and since we already did similar things in one-off projects, I thought it would be good time to look a bit closer on how binfmt_misc interpreter magic works so we can make this a trivial experience every pantahub user.

Let's take a look ...

binfmt_misc registrations are global

After some reading I found that the kernel does not care about the namespace for finding the binfmt_misc definitions (meaning there is only one global knowledge about binfmt_misc wrappers).

If you have permissions you can mount in every container the special binfmt_misc filesystem and you will always see the same, e.g.

# on my system its already mounted
$ mount | grep binfmt_misc
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)

# and loaded
$ ls /proc/sys/fs/binfmt_misc
python3.7  qemu-mips  qemu-mipsel  qemu-riscv32  qemu-riscv64  qemu-x86_64  register  status

You can see that i have a few qemu magics registered. Lets take a look:

$ cat /proc/sys/fs/binfmt_misc/qemu-x86_64 
enabled
interpreter /usr/local/bin/qemu-x86_64-static
flags: 
offset 0
magic 7f454c4602010100000000000000000002003e00
mask fffffffffffefe00fffffffffffffffffeffffff

A good place to get info what those fields mean you can find on wikipedia actually: https://en.wikipedia.org/wiki/Binfmt_misc

binfmt_misc interpreter lookup is namespace-local

However, as expected, Binfmt_Misc does honor the mount namespace of the calling process when searching for the actual interpreter configured.

This allows you to have different versions of python registered depending on the container you are in etc.

Setting up the binfmt_misc handlers

The magic (literally) for identifying elf binaries for each arch is available in the following qemu script:

https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh

We did a small change to that script so we could limit the architectures registered (as we do really just need arm, x86_64, mips(el) and riscv.

In our script you can also overload the default arch list by setting the qemu_target_list env:

https://gitlab.com/pantacor/pv-platforms/wifi-connect/-/blob/master/files/opt/binfmt-misc/qemu-binfmt-conf.sh

And ... it works. \o/