The Benchmarking Puzzle

Benchmarking is hard.

The world is crazy, distributed, diverse, and hard to isolate.

Let's explore why through some common benchmarking scenarios.

Benchmarking Databases

When benchmarking a Database, what do you want to test?

• Write, Read, or Aggregation operations?
• Server:
  • Will you use a single server?
  • How much RAM?
  • What kind of processor?
• Application:
  • Will the application live under the same server?
    • No? So the latency will be 0, which won't be true for 99% of real-world scenarios.
    • Yes? How much latency? How to make sure latency spikes won't mess up the results?

Benchmarking Web Servers

When benchmarking Web Servers, what do you want to test?

• Performance metrics:
  • CPU/RAM Usage?
  • I/O ops/s?
  • p99 latency?
• Language vs Language:
  • Will you use any framework?
    • No?
      • Well, usually it's not a bad idea to use a framework for real-world apps.
    • Yes?
      • Which version of the framework?
      • You know that different versions will probably generate different outcomes, right?
• Server specifications:
  • x86 or arm-based processor?
  • Is there enough RAM?
    • Should it have?
  • Does the processor have 1-core or multi-core?
    • Are you sure the web app is using all the available cores?
      • If you're using JS, you better be sure!
• Benchmark focus:
  • Is it an I/O benchmark or CPU-intensive benchmark?
• Database considerations:
  • What database are you going to use?
  • You know that it makes ALL the difference, right?

Testing

There are so many factors that choosing what to test can be as challenging as crafting the benchmark itself. The answer may be: I WANT TO TEST IT ALL. But then, how do you plan to compare ALL the metrics? Bar charts? Line graphs? Scatter plots?

Conclusion

There are so many pieces to decide that it's overwhelming. In the end, you have to make choices that will definitely impact the results of your benchmarks and, eventually, make the benchmark less useful than you would hope it to be when you started benchmarking.