Process Inspection

Last week I was doing (remote) computer security training with some of my colleagues from work. One of the lab workshops on Windows introduced us to a tool called ProcessHacker. This uses the Windows API to give you finer grain control of Windows processes than is available with the tools provided by default.

One of the features we looked at was the ability to inspect the memory of another process. This uses the ReadProcessMemory API function and is especially useful if you are looking at a process that you suspect should not be on your system.

However, the majority of our work is done using Linux systems, and so one of my colleagues wondered if it was possible to inspect memory from a Linux process in a similar way.

This reminded me of the information available in the /proc filesystem on Linux. I learned about this filesystem many years ago while learning to write kernel modules, and was really intrigued at how so much process instrumentation was made available in userspace using the standard file metaphor. This is why I like *nix style systems so much.

Given what is available under /proc I figured that I should be able to give my colleague what he was looking for. Once I was done, I thought it might be a useful exercise to document.

/proc/${id}/maps

The first step in this was to pick the process that the admin on a system might want to inspect. If someone were trying to do something nefarious, then the first thing to look for might be the memory for a shell. Shells like bash will save their history to a file when they exit, but a user could always try something like:
nohup after 5000 rm ~/.bash_sessions/*
just before logging out. So as an admin, you may want to see what a suspicious user is doing right now, and not after they log out.

Suspect User

So that I'd have something to look for, I used su - testuser to get into a user account, and typed the command: sudo cp /bin/su /tmp/echo
This gave me a bash session belonging to testuser with something I shouldn't see in the history.

Finding a user's process is done with the ps command. I first learned this command with BSD-style options, and because Linux supports these I never really looked at the others. But there are lots of options and lots of output format. Try a man ps and you'll see lots of details.

By default, the ps command shows the processes attached to the current terminal. This is usually just your current shell, and the ps command itself. You can ask to see all the commands for the current user, regardless of the terminal, by adding u to this command. You can also ask to see those commands that are not attached to any terminal by adding x. But the option we really want here is the ability to see commands that belong to all other users, which is a. Since I can usually find what I'm looking for with that information, I default to typing ps aux, and then I will look up a man page only if I need something else.

From the session belonging to testuser I typed ps and got:

  PID TTY          TIME CMD
10852 pts/0    00:00:00 bash
11874 pts/0    00:00:00 ps

So the process ID (PID) that I'm looking for is 10852. Back at a root prompt I went with my standard ps aux and got a very long listing, which included:

root     10851  0.0  0.5  50708  2944 pts/0    S    04:54   0:00 su - testuser
testuser 10852  0.0  1.0  23360  5364 pts/0    S+   04:54   0:00 -su

That's me executing the su - testuser to log in as that user, and the -su is the label I'm seeing for the bash shell that su started for me. There are various ways that a user can get to a shell (such as through ssh), but for now I've found the PID, and can work from there.

So what does the system have on that process?

$ cd /proc/10852
$ ls
attr             cpuset   limits      net            projid_map  stat
autogroup        cwd      loginuid    ns             root        statm
auxv             environ  map_files   numa_maps      sched       status
cgroup           exe      maps        oom_adj        schedstat   syscall
clear_refs       fd       mem         oom_score      sessionid   task
cmdline          fdinfo   mountinfo   oom_score_adj  setgroups   timers
comm             gid_map  mounts      pagemap        smaps       uid_map
coredump_filter  io       mountstats  personality    stack       wchan

There are lots of interesting things in here, and I encourage people to have a look at the docs for /proc for more details. But the data I'm looking for is in the process memory.

mem

The mem file actually represents the process memory. If we look at the entry more carefully, we see:

$ ls -l mem
-rw------- 1 testuser testuser 0 May  1 15:25 mem

So it appears to be a file with no size that belongs to the user who owns the process. It is also only accessible to the process owner, but as root we can inspect it as well.

The file appears to have no size, but that is because it is virtual, and will access the memory as required. However, we can't just access this memory blindly, as it represents addresses that may not be used by the process. To find where data is, we need a map.

maps

The maps virtual file is world readable:

-r--r--r-- 1 testuser testuser 0 May  1 15:25 maps

This file provides instrumentation on how memory is mapped into the process:

$ cat maps
...lots of data...

There is a lot of data here, but what does all this mean? It has a lot to do with how operating systems handle process memory.

Memory Allocation

When a Linux process requires memory, it will use a library function like malloc(3). This typically asks the operating system for more memory by using the brk system call, which it then manages locally. The library function gives the user code the memory it asked for, and then reclaims it for reuse when the memory is no longer needed.

However, memory obtained with brk call is permanently associated with the process, which may not be desired, such as when temporarily handling large buffers. When malloc is asked for very large blocks of data, it instead uses the mmap(2) function. To explain this call, we should look at virtual memory.

Virtual Memory

Back in the days of early personal computing, if a computer program referred to a memory address, such as 0x800 (2048), then this was a precise location in a particular memory chip of the computer. As computing became more powerful and applications demanded more, then it became possible for programs to require more memory than was available on the hardware, and for computers to run more than one program at a time. To manage this virtual memory was created. This uses a combination of the hardware and the operating system to convert arbitrary memory addresses into a physical address that holds the required data.

There are several benefits to this. The first is that data can be moved from physical RAM and onto external media (usually a disk). Then, when the data is needed again, it can be re-loaded back into memory and the address translation will now refer to this new location. As a result, the computer can use some of the disk to behave as if it has more memory than is physically present (albeit, at the cost of speed). And because this translation is done on a process-by-process basis, it also means that processes can use the same addresses to refer to their data, without the risk of accessing the data from another process. This means that one process on a computer is unable to interfere with the operation of another process... at least, not without the intervention of the operating system.

Data gets moved around by an entire "page" at a time. This can be varied, but on Intel hardware the default is 4KiB. (4096 bytes).

Consequently, all memory addresses that a process sees are run through this translation process. If a process refers to an address, then one of 3 things can happen:

• The address is translated through tables to refer to a page located in physical memory. The data at the offset in the page is accessed.
• The address is translated through tables and the page is found to have been "swapped out". Everything pauses for a moment, while the OS reads that page from disk into an available page in physical memory, and an entry for that page is placed into the address translation tables. The data at the offset in the page is then accessed.
• The address is not found in the tables. This is an error.

With this mechanism, the process can act as if it can access any memory from 0 to 2³² or 2⁶⁴ (depending on the hardware and operating system architecture). Of course, only those addresses that have been correctly set up (e.g. by allocating memory) will work.

Most of this is hidden from users, but it can be interacted with using mmap on Linux, or CreateFileMapping or VirtualAlloc on Windows. Both mmap and CreateFileMapping allow you to specify a file descriptor (or file handle for Windows) which permits the OS to allocate virtual memory for you to use, where the locations on disk are provided by the file associated with the handle. Alternatively, on Linux mmap can specify that it is not using a file descriptor, which means that the allocated memory will be backed by the swap file of the operating system. Windows provides similar functionality by calling CreateFileMapping using a parameter of INVALID_HANDLE_VALUE_, or just by calling VirtualAlloc.

System Pages

Using mmap or CreateFileMapping is how large blocks of memory can be set up by the process (possibly referencing files), but what about access to memory that contains library functions? From the perspective of the process, all library functions (including basic i/o functionality like printing text or file access) must appear within its address space.

This is handled in a very similar way to when the process calls mmap or CreateFileMapping. In this case, any libraries that the process needs will be mapped into memory and added to the virtual tables for the process. Most of the central libraries will already be in the system due to being used by other processes, so the only work that needs to be done here is to ensure that their addresses into the virtual memory tables for that process. For many libraries, this initialization is done at process launch. Indeed, the process itself gets launched by mem-mapping the executable file, and then the primary thread is pointed at the initialization address within those pages.

Process Memory Map

Once a process is mapped in, all of its libraries are mapped in, and any additional memory requests have been fulfilled, then various address ranges will represent all of these different parts.

Let's look at the output again...

00400000-004f4000 r-xp 00000000 08:01 393218 /bin/bash
006f3000-006f4000 r--p 000f3000 08:01 393218 /bin/bash
006f4000-006fd000 rw-p 000f4000 08:01 393218 /bin/bash
006fd000-00703000 rw-p 00000000 00:00 0
01b48000-01d16000 rw-p 00000000 00:00 0 [heap]
7f40d6af8000-7f40d6b03000 r-xp 00000000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6b03000-7f40d6d02000 ---p 0000b000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6d02000-7f40d6d03000 r--p 0000a000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6d03000-7f40d6d04000 rw-p 0000b000 08:01 393362 /lib/x86_64-linux-gnu/libnss_files-2.21.so
7f40d6d04000-7f40d6d0e000 r-xp 00000000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6d0e000-7f40d6f0e000 ---p 0000a000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6f0e000-7f40d6f0f000 r--p 0000a000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6f0f000-7f40d6f10000 rw-p 0000b000 08:01 393369 /lib/x86_64-linux-gnu/libnss_nis-2.21.so
7f40d6f10000-7f40d6f25000 r-xp 00000000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d6f25000-7f40d7124000 ---p 00015000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d7124000-7f40d7125000 r--p 00014000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d7125000-7f40d7126000 rw-p 00015000 08:01 393345 /lib/x86_64-linux-gnu/libnsl-2.21.so
7f40d7126000-7f40d7128000 rw-p 00000000 00:00 0
7f40d7128000-7f40d712f000 r-xp 00000000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d712f000-7f40d732e000 ---p 00007000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d732e000-7f40d732f000 r--p 00006000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d732f000-7f40d7330000 rw-p 00007000 08:01 393347 /lib/x86_64-linux-gnu/libnss_compat-2.21.so
7f40d7330000-7f40d74ca000 r-xp 00000000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d74ca000-7f40d76ca000 ---p 0019a000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d76ca000-7f40d76ce000 r--p 0019a000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d76ce000-7f40d76d0000 rw-p 0019e000 08:01 393300 /lib/x86_64-linux-gnu/libc-2.21.so
7f40d76d0000-7f40d76d4000 rw-p 00000000 00:00 0
7f40d76d4000-7f40d76d6000 r-xp 00000000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d76d6000-7f40d78d6000 ---p 00002000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d78d6000-7f40d78d7000 r--p 00002000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d78d7000-7f40d78d8000 rw-p 00003000 08:01 393331 /lib/x86_64-linux-gnu/libdl-2.21.so
7f40d78d8000-7f40d78fe000 r-xp 00000000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d78fe000-7f40d7afd000 ---p 00026000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d7afd000-7f40d7b01000 r--p 00025000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d7b01000-7f40d7b02000 rw-p 00029000 08:01 395088 /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f40d7b02000-7f40d7b24000 r-xp 00000000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7b24000-7f40d7d23000 ---p 00022000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7d23000-7f40d7d24000 r--p 00021000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7d24000-7f40d7d25000 rw-p 00022000 08:01 395006 /lib/x86_64-linux-gnu/libncurses.so.5.9
7f40d7d25000-7f40d7d47000 r-xp 00000000 08:01 393292 /lib/x86_64-linux-gnu/ld-2.21.so
7f40d7db5000-7f40d7df4000 r--p 00000000 08:01 662823 /usr/lib/locale/zu_ZA.utf8/LC_CTYPE
7f40d7df4000-7f40d7f24000 r--p 00000000 08:01 662822 /usr/lib/locale/zu_ZA.utf8/LC_COLLATE
7f40d7f24000-7f40d7f28000 rw-p 00000000 00:00 0
7f40d7f33000-7f40d7f34000 r--p 00000000 08:01 662821 /usr/lib/locale/zu_ZA.utf8/LC_NUMERIC
7f40d7f34000-7f40d7f35000 r--p 00000000 08:01 674657 /usr/lib/locale/en_US.utf8/LC_TIME
7f40d7f35000-7f40d7f36000 r--p 00000000 08:01 674656 /usr/lib/locale/en_US.utf8/LC_MONETARY
7f40d7f36000-7f40d7f37000 r--p 00000000 08:01 673013 /usr/lib/locale/ug_CN/LC_MESSAGES/SYS_LC_MESSAGES
7f40d7f37000-7f40d7f38000 r--p 00000000 08:01 672854 /usr/lib/locale/yi_US.utf8/LC_PAPER
7f40d7f38000-7f40d7f39000 r--p 00000000 08:01 672855 /usr/lib/locale/yi_US.utf8/LC_NAME
7f40d7f39000-7f40d7f3a000 r--p 00000000 08:01 674655 /usr/lib/locale/en_US.utf8/LC_ADDRESS
7f40d7f3a000-7f40d7f3b000 r--p 00000000 08:01 672853 usr/lib/locale/yi_US.utf8/LC_TELEPHONE
7f40d7f3b000-7f40d7f3c000 r--p 00000000 08:01 672852 /usr/lib/locale/yi_US.utf8/LC_MEASUREMENT
7f40d7f3c000-7f40d7f43000 r--s 00000000 08:01 288093 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f40d7f43000-7f40d7f44000 r--p 00000000 08:01 674654 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
7f40d7f44000-7f40d7f46000 rw-p 00000000 00:00 0
7f40d7f46000-7f40d7f47000 r--p 00021000 08:01 393292 /lib/x86_64-linux-gnu/ld-2.21.so
7f40d7f47000-7f40d7f48000 rw-p 00022000 08:01 393292  /lib/x86_64-linux-gnu/ld-2.21.so
7f40d7f48000-7f40d7f49000 rw-p 00000000 00:00 0
7ffcd6454000-7ffcd6475000 rw-p 00000000 00:00 0 [stack]
7ffcd654f000-7ffcd6551000 r--p 00000000 00:00 0 [vvar]
7ffcd6551000-7ffcd6553000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

The first 3 lines all show address ranges written in hexadecimal, along with some other data:

00400000-004f4000 r-xp 00000000 08:01 393218 /bin/bash
006f3000-006f4000 r--p 000f3000 08:01 393218 /bin/bash
006f4000-006fd000 rw-p 000f4000 08:01 393218 /bin/bash

Hexadecimal is very useful for representing binary data, since each digit is exactly represented by 4 bits, whereas decimal needs a calculator to convert back and forth. The first of these addresses describes addresses in the range 4194304-5193728. This is a block of size 0xf4000, or 999424 bytes.

These numbers seem arbitrary, so let's consider them in pages of 4k. 4096 is 0x1000 in hex, so the addresses range from page number 0x400 to page number 0x4f4, for a range of 0xf4 pages. That makes the numbers look much smaller and more regular. 😊

The next block starts at page 0x6f3 and goes up to 0x6f4, but is immediately followed by the next block which starts at that ending address, and goes up to 0x6fd. So together, this makes 10 4k pages (or 0xa pages).

Honestly, I don't really care about the size or ranges of these addresses. What I really notice is that the map explicitly connects these pages to the file /bin/bash. They also show an offset into the file where the mapping starts at. There is some overlap between these mapping, and the reason can be seen when the page properties are considered. The first mapping (covering the first 0xf4 pages of the file) is marked as executable. This means that it contains runnable code. The next mapping is marked as read only, meaning that any access to those pages should be for reading data. I would expect to find static definitions in there, like message strings. The final block is marked as read/write. This might include statically initialized data that can be modified by the running processes.

Further down, we see lots of references to mappings of the shared libraries: the .so files. For instance, libc-2.21.so contains the standard libraries for the language C (which bash was written in), and libncurses.so.5.9 contains the ncurses library which is used for creating text interfaces.

Then we have 2 more types of page ranges. The first type includes the 6 ranges that have no file path information. These were presumably given to the process when code requested memory to be allocated, and will likely contain data structures. The second type includes labels rather than a file path. Of particular interest to this process are the stack and the heap.

For those who are unfamiliar with these elements of memory layout, the stack is a memory structure used by the computer to move from one function to another. Whenever a function is called, the computer adds data that represents arguments to the "top" of the stack, adds a pointer to the last "top" of the stack along with the current execution address all at the top of the stack, and then jumps to the new function. That function can now see its arguments near the top of the stack. Any variables it needs to create temporarily can also be added to the top of the stack. Once the function is done, the stack is read to find where the previous top-of-stack was along with the previous execution address, and these get restored.

This setup is how code can call into functions, and when the function is finished, it can "return" and the calling code continue where it left off. The data structure is often likened to a stack of plates, where plates can be added to the top, and then removed again after.

Now that we know what the stack is, we can see that it will be rapidly changing data that contains internal data structures representing the function calls.

The heap, on the other hand, is where a program gets memory that it requests with malloc. This is longer-term working memory. It will typically contain internal data structures, but if strings are being stored by a program, then this is where we would expect to find them. In this case, I'm hoping to find the history of the process, so I want to look in here.

Pulling Out the Heap

The relevant line for the heap is:

00f5d000-0111d000 rw-p 00000000 00:00 0 [heap]

This means that we want to see memory starting at address 0xf5d000 and ending 0x1c0000 later at address 0x111d000. Dividing by 4k pages, this starts at the 0xf5d page, and extends for 0x1c0 pages.

We've already talked about how we can access memory through the mem virtual file, and it just so happens that we have the dd tool that can read precise portions of a file. So we just need to establish the arguments for it.

status

The dd command likes to describe how many blocks it read, how many it wrote, and things like that. We don't want to see that, so we can set the status value to none.

file

The file to be read will be mem, but to access it from elsewhere, we want to refer to the file as /proc/10668/mem. This can generalized to read from an arbitrary pid variable with: /proc/${pid}/mem

ibs

It may be easier to ask dd to handle data a byte at a time, but we already know that the data comes in 4k pages, so it will be much more efficient to read the memory "file" in 4k blocks. This means that we want to set the "input block size" or ibs to 4096.

skip

We also want to skip over all the invalid addresses right up to the start of the heap pages. That means skipping 0xf5d pages. However, like ibs, we need to use decimal for the dd command. In this case, that's 3933 pages.

count

The number of blocks to read is the final address, minus the first address, divided by the block size. That's 0x111d000 - 0xf5d000, which gives, 0x1c0000. Dividing by 0x1000 (4096) gives 0x1c0, which is 448 in decimal.

This gives us a final command line of:

dd status=none if=/proc/${pid}/mem ibs=4096 skip=3933 count=448

The data coming out can be saved or piped into another program for processing.

Automating

This was fine for my system, but there was lots of calculating to get the values necessary to execute this command. I couldn't pass it along to my colleague in that form.

I decided that I could provide a function that takes an argument of the desired process ID to scan. As above, I'll refer to this as $pid. So that the PID can be placed in the middle of a string, we can wrap it with braces: ${pid}

Range

The address range for the heap can be found by using grep to search for the string "heap" in the maps file.

grep heap /proc/${pid}/maps
00f5d000-0111d000 rw-p 00000000 00:00 0 [heap]

That returns the entire line, but we just want the first part of the line. We can split the line up using the cut command, telling it to separate fields on the space character, and then saying that we just want the first field.

grep heap /proc/${pid}/maps | cut -d' ' -f1
00f5d000-0111d000

We can save this result to the variable $range by wrapping everything in backticks:

range=`grep heap /proc/${pid}/maps | cut -d' ' -f1`

Arithmetic

Luckily for us, bash has several built in features that we need right now: text substitution, hex/decimal conversion, and arithmetic.

To get the start of the range, we want everything in the range up to the first - character. We can do that by saying that we want to substitute a dash followed by any characters with an empty string. The syntax for this is: ${range/-*/}

The second part of the range can be found by substituting "a series of characters followed by -" with an empty string: ${range/*-}

Once we have string representations for the numbers, we can do arithmetic by wrapping expressions with $(( )).

Arithmetic needs to be done in decimal, but once we are inside an arithmetic expression, we can print any hexadecimal (base 16) number as decimal by preceding the value with 16#:

$((16#1c0))
448

So now we can build an expression for each of the dd parameters, by extracting the value, converting to decimal, and dividing by 4096:

# skip
$((16#${range/-*/}/4096))
# count
$(((16#${range/*-/}-16#${range/-*/})/4096))

The full command is then:

dd status=none if=/proc/${pid}/mem ibs=4096 skip=$((16#${range/-*/}/4096)) count=$(((16#${range/*-/}-16#${range/-*/})/4096))

Processing

The dd command pulls out the heap, but this is binary data for an active process, and should not be printed. Data structures could be inspected using a tool like hexdump but in this case I'm looking for text in the user's history. The best tool for this is the strings command, which filters out anything that is not printable data.

Final Function

Here is the final function:

I can run it against the testuser process, but I'm looking for something suspicious done with sudo, so let's look for that:

$ string_heap 10852 | grep sudo
sudo
sudo cp /bin/su  /tmp/echo
sudo cp /bin/su  /tmp/echo
sudo cp /bin/su  /tmp/echo
/usr/share/bash-completion/completions/sudo
_sudo
sudo
_sudo
sudoedit
_sudo
_sudo
/usr/bin/sudo
sudo cp /bin/su /tmp/echo
*sudoedit
_sudo
/etc/init.d/sudo
gksudo
kdesudo
_sudo
*sudoedit
_sudo
_=/usr/bin/sudo

Sure enough... there is the command at the top of the output.

Recap

This was a very short description of some of the instrumentation provided in the Linux /proc filesystem, in relation to process memory. We also discussed virtual memory, its allocation, and usage in Linux and Windows. Finally, I showed how to write a short bash script that can scan the memory of another process and look for suspicious activity.