Benchmark Design

Benchmark design

A benchmark is an experiment, and most of them are broken experiments that measure the JIT warming up, the allocator settling, or the dead-code eliminator deleting the thing under test. A confident wrong number is worse than no number, because someone ships a decision on it. Design the measurement so the result survives being run again on another machine.

Method

1. State the question and the unit first. "Nanoseconds per parse() call on 1 KB inputs", not "is it fast". Fix the input size, the data shape, and what counts as one operation before writing any timing code.
2. Warm up until steady state, then measure. Run the workload untimed until the JIT compiles and caches fill (JMH @Warmup, Criterion and pytest-benchmark do this automatically). Discard warmup samples. A cold first run measures compilation, not the code.
3. Defeat dead-code elimination. Consume every result: return it, feed it to a blackhole (Blackhole.consume in JMH), or accumulate into a checksum you print. A loop whose output is unused legally compiles to nothing, and then you are timing an empty loop.
4. Isolate the machine. Pin to a core (taskset -c 2), disable turbo and frequency scaling (cpupower frequency-set -g performance), close background load, and run on hardware, not a shared CI runner or a laptop on battery. Report the CPU, OS, and runtime version alongside the number.
5. Report a distribution, not a mean. Collect many samples and show median plus p95 and the interquartile range. The mean hides multimodal results from GC pauses and context switches; the median with spread tells the truth. Flag any run whose variance exceeds a few percent as untrusted.
6. Compare with a real significance test. For A versus B, run interleaved samples and apply a Mann-Whitney U test or non-overlapping confidence intervals, not eyeballed averages. A 3% difference inside the noise band is not a difference.

Litmus tests

• Does re-running the benchmark on a second machine preserve the ranking of the variants, even if absolute numbers shift?
• If you delete the code under test, does the benchmark get faster (it should break or error, proving the work was actually happening)?
• Do you report median and spread with hardware and runtime, not a lone mean?
• Would the measured difference survive a significance test on fresh samples?

Boundaries

This governs microbenchmarks and small controlled comparisons. Whole-system load behavior under concurrency belongs to throughput-scaling and latency-analysis; catching drift over time in automation is performance-regression-detection. Production numbers come from real-user-monitoring, not a bench.