Performance testing
A benchmark that reports one number from one run is worse than no benchmark:
it invites decisions made from noise. Honest measurement accounts for warmup,
background variance, and input shape, and it states its uncertainty. Skip the
method and you end up optimizing against your cache state and a neighbor's
cron job rather than the code.
Method
- Warm up before you measure. The first iterations pay for JIT
compilation, cold caches, and lazy allocation. Run a discard loop until
timings stabilize, then start recording. Established harnesses like JMH,
pytest-benchmark, and Criterion do this for you; respect it.
- Fix the input and the input size. Benchmark a known, committed dataset
at a stated size, not whatever happened to be in memory. Report the size
with the result: "parse, 10k rows" means something, "parse is fast" does
not.
- Run many iterations and report the distribution. Collect hundreds of
samples and report median plus a spread such as p95 and standard
deviation. The mean alone hides a bimodal result where half the runs hit a
slow path.
- Quiet the machine. Close other load, pin to a core, and disable
frequency scaling or turbo boost where you can. On a laptop on battery the
numbers are fiction.
hyperfine and taskset help isolate a run.
- Compare against a committed baseline, not a memory. Store last known
timings in the repo and diff against them. "20% slower than the baseline
we committed" is actionable; "feels slower than last week" is not.
- Gate regressions in CI with a threshold and repeats. Fail the build
when the median exceeds the baseline by a set margin, say 10%, confirmed
over repeated runs so a single noisy sample does not block a merge. Track
the baseline as it legitimately improves.
- Profile to find the cost, benchmark to confirm the fix. A profiler
like perf, py-spy, or a flame graph tells you where time goes; the
benchmark proves the change actually moved the median, not just your
intuition.
Signals
- Do two runs of the unchanged benchmark land within your CI threshold of
each other? If not, the harness is too noisy to gate on.
- Does the report name the input size, iteration count, and a spread, not
just a single millisecond figure?
- When the gate fails, can you point to the committed baseline it regressed
against?
Boundaries
This covers micro and function-level benchmarks on one machine. Multi-user
capacity, ramp profiles, and saturation belong to load-testing. Whether a
given speed is fast enough is a product call against a latency budget, not
something a benchmark decides.
1---2name: performance-testing3description: Benchmark code so the numbers mean something: warm up, fix inputs, report variance, and gate regressions in CI. Use when measuring the speed of a function or comparing two implementations.4---56# Performance testing78A benchmark that reports one number from one run is worse than no benchmark:9it invites decisions made from noise. Honest measurement accounts for warmup,10background variance, and input shape, and it states its uncertainty. Skip the11method and you end up optimizing against your cache state and a neighbor's12cron job rather than the code.1314## Method15161. **Warm up before you measure.** The first iterations pay for JIT17 compilation, cold caches, and lazy allocation. Run a discard loop until18 timings stabilize, then start recording. Established harnesses like JMH,19 pytest-benchmark, and Criterion do this for you; respect it.202. **Fix the input and the input size.** Benchmark a known, committed dataset21 at a stated size, not whatever happened to be in memory. Report the size22 with the result: "parse, 10k rows" means something, "parse is fast" does23 not.243. **Run many iterations and report the distribution.** Collect hundreds of25 samples and report median plus a spread such as p95 and standard26 deviation. The mean alone hides a bimodal result where half the runs hit a27 slow path.284. **Quiet the machine.** Close other load, pin to a core, and disable29 frequency scaling or turbo boost where you can. On a laptop on battery the30 numbers are fiction. `hyperfine` and `taskset` help isolate a run.315. **Compare against a committed baseline, not a memory.** Store last known32 timings in the repo and diff against them. "20% slower than the baseline33 we committed" is actionable; "feels slower than last week" is not.346. **Gate regressions in CI with a threshold and repeats.** Fail the build35 when the median exceeds the baseline by a set margin, say 10%, confirmed36 over repeated runs so a single noisy sample does not block a merge. Track37 the baseline as it legitimately improves.387. **Profile to find the cost, benchmark to confirm the fix.** A profiler39 like perf, py-spy, or a flame graph tells you where time goes; the40 benchmark proves the change actually moved the median, not just your41 intuition.4243## Signals4445- Do two runs of the unchanged benchmark land within your CI threshold of46 each other? If not, the harness is too noisy to gate on.47- Does the report name the input size, iteration count, and a spread, not48 just a single millisecond figure?49- When the gate fails, can you point to the committed baseline it regressed50 against?5152## Boundaries5354This covers micro and function-level benchmarks on one machine. Multi-user55capacity, ramp profiles, and saturation belong to load-testing. Whether a56given speed is fast enough is a product call against a latency budget, not57something a benchmark decides.
Amey-Thakur