Tracing msgrcv with ltrace

ltrace is a tool for tracing dynamic library calls. I use it from time to time and this time I needed to trace the msgrcv system call. Usually, you would use strace to trace the system calls, but I need an output of both library calls and system calls. One interesting feature is that I don’t need to trace the actual system calls using the -S flag and can trace the libc system call wrappers instead:

msgrcv@libc.so.6(0x10002, 0x5634e4d2db08, 4712, 0xffffffffc0000000) = 708 <0.828835>

So far so good, but the output is not very useful as I don’t see the message received. ltrace properly displays the most common library calls but for some reason not the msgrcv and several others like semop and shmat. Luckily for me ltrace can use function prototype descriptions from a custom ltrace.conf file. msgrcv is a tricky function to describe: just like read and similar functions, it uses output parameter and the length of the data is returned by the function. In addition to that, the output parameter is a complex type. My first attempt at configuration got me pretty far:

long msgrcv(int, +struct(long, array(char, retval))*, ulong, long, hex(int));

An important detail is the + sign before the output parameter. ltrace will wait until the function returns and will display the rest of the parameters after that. The second important detail is specifying the length of the data as retval (an alias for arg0). It is the value function returned. I could have used the arg3 and show the full content of the second argument but using the function result allows to show only the portion of the array that is filled:

msgrcv@libc.so.6(65538, { 536870912, ['{', '\0', '\0', '\0', '\002'] }, 4712, -1073741824, 0) = 5 <19.964097>

I was very happy until the msgrcv system call failed. The function returned a -1 which was interpreted as an unsigned integer and ltrace printed a huge amount of junk until it hit the limit of array size to be printed. Turns out this issue is described in the “to do” list of the ltrace for many years without any progress. But the same “to do” revealed that there is a workaround. I can wrap the retval in zero() and the result will be much better:

long msgrcv(int, +struct(long, array(char, zero(retval)))*, ulong, long, hex(int));

The only downside is that I get an error message for failing calls, but it is much easier to deal with than kilobytes of junk:

msgrcv@libc.so.6(3833859 <no return ...>
error: maximum array length seems negative
, { 0, ['{'] }, 4712, 0, 0x800) = -1 <0.000520>