CPU Time: How To Accurately Benchmark Your Code

I'm going to assume that everyone reading this post has done some benchmarking here and there. But have you ever thought about how accurate are your readings?

In this post, we will talk about a few misconceptions, task schedulers, CPU timings, and a real example in C.

Misconceptions

Time matters

Ok, to be fair, the title is click-bait. Time really matters but not the time you see on your clock! We only care how much time is CPU spending on our process and that's usually not the wall-clock time.

Slow software = Slow code

The speed of your software depends on the environment that's running. I'm not talking about your computer specs. Softwares rely heavily on syscalls or I/O interfaces.
During a benchmark, You really should pay attention to this subject since I/O is really really slow and I've seen too many times when software is considered slow without realizing that a huge part of this problem is caused by slow I/O and can be improved with technics like caching and etc.

Performance is not important

Performance is UX but most micro-optimizations are usually time-wasting and not worth it. I think there's a fine line that we shouldn't cross when developing a product. You can pack your product with features but most people will hate it at a glance if It's too slow or sluggish.
I'd say that finding this line may not be as easy as saying it but I don't see another way around it.

Schedulers

With the notion of multi-tasking operating systems, it came task schedulers. To be able to run multiple programs at the same time we need a "scheduler" to schedule the work between limited CPU cores.

We will discuss two types of schedulers.

Preemptive Multitasking

In this model of multitasking, the kernel preemptively schedules tasks and interrupts processes to let other processes run.

Our software doesn't notice these interrupts since its state is frozen to be used later when there's free CPU time for the process.

The Linux kernel scheduler is an example of this model.

Cooperative Multitasking

Unlike preemptive multitasking, this model relies solely on the code itself to free the resources back to the scheduler.

So the scheduler itself has no control over the process until the process yields back to let the scheduler know that the job is done or It's blocked so the CPU can be used for other processes.

The Go language goroutine scheduler is an example of this model.

CPU Time

CPU Time is the time that is actually spent on the CPU. So no I/O time is included in this.

For example, if you're doing HTTP requests and each request takes a second to respond, the CPU time doesn't include that second since during that time, we didn't do any CPU work and just waited for the response.

In Linux, A POSIX compliant kernel, these times are kept for each process or thread.

Code

In this very simple program, we use the clock() function to benchmark a simple pi calculation code.

#include <time.h>

// https://crypto.stanford.edu/pbc/notes/pi/code.html
int pi_calculate()
{
    int r[2800 + 1];
    int i, k;
    int b, d;
    int c = 0;

    for (i = 0; i < 2800; i++)
    {
        r[i] = 2000;
    }

    for (k = 2800; k > 0; k -= 14)
    {
        d = 0;

        i = k;
        for (;;)
        {
            d += r[i] * 10000;
            b = 2 * i - 1;

            r[i] = d % b;
            d /= b;
            i--;
            if (i == 0)
                break;
            d *= i;
        }
        c = d % 10000;
    }

    return 0;
}

int main(void)
{
    clock_t start_time = clock();
    int pi;

    for (int i = 0; i < 100; i++)
    {
        pi = pi_calculate();
    }

    double elapsed_time = (double)(clock() - start_time) / CLOCKS_PER_SEC;
    printf("%f seconds\n", elapsed_time);
}

A few relevant and useful functions:

clock()

#include <time.h>

clock_t clock(void);

This is the function that we used in our code to get the clock time of our current process.

clock_getcpuclockid()

#include <time.h>

int clock_getcpuclockid(pid_t pid, clockid_t *clockid);

This function returns a CPU clock id by using a PID. By obtaining the clock id you're able to get the CPU usage.

As you can see, you can easily get the time of a clock.

pthread_getcpuclockid()

#include <pthread.h>
#include <time.h>

int pthread_getcpuclockid(pthread_t thread, clockid_t *clockid);

If you're working in multi-thread software, You may use this function to get the clock of a POSIX thread.

clock_gettime()

#include <time.h>

int clock_gettime(clockid_t clockid, struct timespec *tp);

struct timespec {
     time_t   tv_sec;        /* seconds */
     long     tv_nsec;       /* nanoseconds */
};

Use this function to get the time of a clock.

Conclusion

In this post, we went through a few terminologies regarding the schedulers and timing and then learned how to accurately benchmark our code using the CPU clock time that kernel provides for us.

I really hope you enjoyed this post.

Links

• https://en.wikipedia.org/wiki/CPU_time
• https://crypto.stanford.edu/pbc/notes/pi/code.html
• https://man7.org/linux/man-pages/man3/pthread_getcpuclockid.3.html
• https://man7.org/linux/man-pages/man3/clock_getcpuclockid.3.html
• https://man7.org/linux/man-pages/man3/clock.3.html
• https://man7.org/linux/man-pages/man2/clock_gettime.2.html