bench-protocol
Rules
- Interleave. Run the arms in rounds with rotating order (A B, B A, ...). Never run
all of A and then all of B. Machine state (thermal, turbo, co-tenants, page cache)
drifts. Absolute times can swing by a factor of two across hours on a shared
server, so only ratios from back-to-back pairs in the same session are trusted.
- Repeat. Use at least 3 measured repeats plus at least 1 discarded warm-up round.
Use at least 7 when the arms differ by less than 10% or the spread is large. Single
measurements of the same binary can differ by nearly 2×.
- Report the median, with the IQR and a bootstrap 95% CI. For a ratio, report
median(A)/median(B) with a paired bootstrap CI. Never report the mean of a skewed
timing distribution as the headline number. Never report the best of N unless the
protocol states it for both arms.
- Lock and isolate. Hold
crbench's file lock so benches never overlap. Run
benches one at a time with low concurrency: parallel benches compete for memory
bandwidth and inflate the arm that uses it more. Keep the machine free of other
heavy jobs. crbench env warns on high load and non-performance governors.
- Control threads. Set
OMP_NUM_THREADS, RAYON_NUM_THREADS, GOMAXPROCS and
the MKL/OpenBLAS thread variables explicitly (--threads). Pin with taskset
(--cpus) when comparing single-thread numbers. Report the single-thread comparison
even when the headline is parallel.
- No stale builds. Build from the pinned commit into a configuration-specific
directory. Run
crbench stale BIN SRC before each campaign. Record the build flags.
- Record the environment.
crbench run stores CPU, governor, load, compilers,
git commit and dirty flag of --target, and the performance environment variables in
run.json. Keep it.
- Record failures. A crash, timeout or missing output stays in the ledger as a
failed execution with its error. It is excluded from statistics but never deleted.
If one arm fails systematically, that is a result: document the boundary.
- Verify correctness in the timed binary. The benchmark must check its output
(decrypt and compare, verify). A fast wrong answer is not a data point.
- Explain surprises. If a wall-clock speed-up exceeds the operation-count ratio,
find the mechanism before publishing. Examples: evaluation at a lower level with a
smaller modulus, lower depth, cache effects. Otherwise, suspect the measurement.
Command
crbench run \
-a base='./build-base/bench --set toy' \
-a new='./build-new/bench --set toy' \
-n 7 -w 1 --order alternate --parser generic --metric time \
--threads 1 --cpus 2 --target ./src --log-dir results/2026-01-01-toy-ab
Outputs: results/.../run.json (all executions, environment, summary),
results/.../<arm>/rNNN.log (raw output of each execution), and a JSON summary on
stdout with the speed-up and its CI.
Choosing the metric
| domain |
primary metric |
also record |
| FHE |
latency per operation, or amortised per slot or bit |
peak memory (/usr/bin/time -v), key sizes, multiplicative depth or level used |
| lattice attacks / estimators |
wall time and success rate over seeds |
block size β, dimension, memory |
| SAT / MILP |
time to solve, median over ≥ 10 seeds or instances |
timeouts (count), solver version, CNF/LP hash |
| MPC / PSI |
end-to-end time per network profile |
bytes sent per party, rounds, offline/online split |
| PQC |
median cycles (fixed frequency) |
stack and heap usage, key and ciphertext/signature sizes |
When numbers disagree across sessions
Rerun both arms back to back in one session. If the new pair contradicts an earlier
headline, the same-session pair wins. Withdraw the old number explicitly in
EVIDENCE.md (strike it through and add a reason) rather than silently replacing it.
Check whether the cache is really the variable before you blame it: run a cache-isolation
experiment.
Hand-off
run.json and logs are in place, with at least 3 successful repeats per arm. Then
use skills/log-to-evidence to create EVIDENCE rows.
1---2name: bench-protocol3description: Measurement protocol for trustworthy timings. It covers interleaved A/B rounds, medians with IQR and bootstrap CIs, at least 3 repeats, warm-up, no stale builds, an isolated and locked machine, thread pinning, and recorded environment. Use for every performance number, whether FHE, lattice attack, SAT/MILP, MPC/PSI or PQC, before it can enter EVIDENCE.md.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/bench-protocol6---78# bench-protocol910## Rules11121. **Interleave.** Run the arms in rounds with rotating order (A B, B A, ...). Never run13 all of A and then all of B. Machine state (thermal, turbo, co-tenants, page cache)14 drifts. Absolute times can swing by a factor of two across hours on a shared15 server, so **only ratios from back-to-back pairs in the same session are trusted.**162. **Repeat.** Use at least 3 measured repeats plus at least 1 discarded warm-up round.17 Use at least 7 when the arms differ by less than 10% or the spread is large. Single18 measurements of the same binary can differ by nearly 2×.193. **Report the median**, with the IQR and a bootstrap 95% CI. For a ratio, report20 `median(A)/median(B)` with a paired bootstrap CI. Never report the mean of a skewed21 timing distribution as the headline number. Never report the best of N unless the22 protocol states it for both arms.234. **Lock and isolate.** Hold `crbench`'s file lock so benches never overlap. Run24 benches one at a time with low concurrency: parallel benches compete for memory25 bandwidth and inflate the arm that uses it more. Keep the machine free of other26 heavy jobs. `crbench env` warns on high load and non-`performance` governors.275. **Control threads.** Set `OMP_NUM_THREADS`, `RAYON_NUM_THREADS`, `GOMAXPROCS` and28 the MKL/OpenBLAS thread variables explicitly (`--threads`). Pin with `taskset`29 (`--cpus`) when comparing single-thread numbers. Report the single-thread comparison30 even when the headline is parallel.316. **No stale builds.** Build from the pinned commit into a configuration-specific32 directory. Run `crbench stale BIN SRC` before each campaign. Record the build flags.337. **Record the environment.** `crbench run` stores CPU, governor, load, compilers,34 git commit and dirty flag of `--target`, and the performance environment variables in35 `run.json`. Keep it.368. **Record failures.** A crash, timeout or missing output stays in the ledger as a37 failed execution with its error. It is excluded from statistics but never deleted.38 If one arm fails systematically, that is a result: document the boundary.399. **Verify correctness in the timed binary.** The benchmark must check its output40 (decrypt and compare, verify). A fast wrong answer is not a data point.4110. **Explain surprises.** If a wall-clock speed-up exceeds the operation-count ratio,42 find the mechanism before publishing. Examples: evaluation at a lower level with a43 smaller modulus, lower depth, cache effects. Otherwise, suspect the measurement.4445## Command4647```bash48crbench run \49 -a base='./build-base/bench --set toy' \50 -a new='./build-new/bench --set toy' \51 -n 7 -w 1 --order alternate --parser generic --metric time \52 --threads 1 --cpus 2 --target ./src --log-dir results/2026-01-01-toy-ab53```5455Outputs: `results/.../run.json` (all executions, environment, summary),56`results/.../<arm>/rNNN.log` (raw output of each execution), and a JSON summary on57stdout with the speed-up and its CI.5859## Choosing the metric6061| domain | primary metric | also record |62|---|---|---|63| FHE | latency per operation, or amortised per slot or bit | peak memory (`/usr/bin/time -v`), key sizes, multiplicative depth or level used |64| lattice attacks / estimators | wall time and success rate over seeds | block size β, dimension, memory |65| SAT / MILP | time to solve, median over ≥ 10 seeds or instances | timeouts (count), solver version, CNF/LP hash |66| MPC / PSI | end-to-end time per network profile | bytes sent per party, rounds, offline/online split |67| PQC | median cycles (fixed frequency) | stack and heap usage, key and ciphertext/signature sizes |6869## When numbers disagree across sessions7071Rerun both arms back to back in one session. If the new pair contradicts an earlier72headline, the **same-session pair wins**. Withdraw the old number explicitly in73EVIDENCE.md (strike it through and add a reason) rather than silently replacing it.74Check whether the cache is really the variable before you blame it: run a cache-isolation75experiment.7677## Hand-off7879`run.json` and logs are in place, with at least 3 successful repeats per arm. Then80use `skills/log-to-evidence` to create EVIDENCE rows.