# Dev Benchmark

> Measuring Capsem with capsem-bench. Use when running benchmarks, adding a dimension or a collector, reading the store, or judging whether a slowdown is real.

- Skill: `google/dev-benchmark` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add google/dev-benchmark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/google/dev-benchmark/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Google (https://skillmd.com/u/google)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/google/dev-benchmark

---


# Benchmarking

One binary computes every number. A collector prints raw samples and computes
nothing. That split is the whole design, and it exists because the thing it
replaced had ten producers each computing its own `p99`.

## Quick start

```bash
just bench-quick                 # dev loop: no guest, bounded, records like any run
just bench                       # every dimension that has a collector
just bench routes criterion      # only these
just bench-report                # what each subject reads, and how it has moved
```

`capsem-bench-rs doctor` first if a number surprises you. Every measurement
this repo took before it existed was taken on an unqualified machine.

## The binary

`crates/capsem-bench`, shipped as `capsem-bench-rs` (host and guest).

| subcommand | does |
|---|---|
| `list` | every dimension and whether the quick lane covers it |
| `run [dim...]` | measure, then record into the store |
| `doctor` | is this machine fit to measure -- load, governor, KVM, strays |
| `compare A B <dim>` | two stores, metric by metric |
| `verify` | ratchet a run against evidence; refuses on an unfit machine |
| `report` | every subject and its trend |
| `protocol`, `protocol-delta`, `delta` | the scenario engine: host-direct vs guest-through-Capsem |

Modules: `schema.rs` (the record), `stats.rs` (**all** statistics),
`store.rs` (SQLite), `collector.rs` (the subprocess protocol), `machine.rs`
(`doctor`), `scenarios.rs` and `protocol.rs` (the scenario engine).

## The record

`capsem.bench.v1`: one envelope, one clock, explicit identity, a flat metric
list with stable dotted keys (`gateway./vms/list.cpu_s`). Per metric: `n`,
`min`, `max`, `mean`, `median`, `p90`, `p95`, `p99`, `p999`, `stddev`, `cv`,
`mad`, rounded to two decimals.

It lives in SQLite -- `cache/target/tests/benchmarks/benchmarks.db`, two tables. The
scheme before it was a file per dimension per release per architecture per
profile in ten incompatible shapes, one of them 80 KB of captured stdout;
asking "is `/vms/list` slower than three releases ago" meant globbing filenames
and knowing which shape each match used. It is now a query.

`quick` runs are recorded and never selected as evidence, so a dev-loop
measurement is visible without becoming a baseline. Architecture and profile
must match before two numbers are compared -- otherwise you have measured the
difference between two machines.

## Collectors

`benchmarks/collectors/<dimension>`, executable, no suffix. Print one JSON document
of raw samples on stdout and nothing else:

```json
{"metrics": {"cpu_s": {"unit": "seconds", "samples": [0.14, 0.15]}}}
```

Rules the collector protocol enforces, each because it was violated:

- a second document is refused -- the collector ran twice and only the first
  would ever be read
- an empty `metrics` map is refused -- it passes every ratchet
- a metric with no samples is refused
- leading noise before the document is tolerated, so a collector can keep its
  own diagnostics
- every collector is bounded; one that never exits holds the machine lock the
  whole gate runs under

`[boundary.bench]` caps a collector at 300 lines. A collector growing past that
means statistics have started living in it again.

The eight guest collectors are one file, `_guest`, symlinked per mode: they
read their mode from `argv[0]` and drive the modules in
`guest/artifacts/capsem_bench/`. `criterion` derives its targets from the Cargo
manifests and reads raw per-iteration nanoseconds out of Criterion's
`sample.json`, so a new `[[bench]]` target is measured without editing it.

## Is the move real?

`compare` and `verify` report `delta_abs`, `delta_pct`, `ratio`, and
`significant`. A bare `ratio > 1.1` cannot tell a regression from a jittery
machine, which is exactly what happened: 0.6.0 qualification failed on
`gateway /vms/list CPU=0.160s > 0.140s`, and a rerun showed it was a one-off.
`significant` requires the breach to also exceed the baseline's own `cv` and
the config-owned minimum time resolution. A percentage on nanoseconds is not
product performance: those deltas remain visible as `tiny`, but never hold a
release.

Treat low-single-digit route-latency movement, including 3-5 ms, as host and
scheduler noise rather than a product speedup or regression. Route-health
ceilings are rounded gross-regression guards with a config-owned 20% operating
margin; they are not comparative performance claims. A real comparative claim
comes from repeated stored evidence and must clear its observed variance.

`[benchmark_regression] maximum_factor` is the ratio, config-owned and
relative to checked-in evidence. `minimum_time_resolution_ms` is a resolution
floor, not a performance budget: the ratio still catches meaningful drift
without pretending sub-millisecond product precision exists. Comparative
performance gets a 10% envelope; separately authored health ceilings retain
20% operating headroom and are rounded to human millisecond/second increments.

## Route coverage

53 of the 101 service routes had no timing signal, and nothing noticed, because
coverage was a hand-maintained list inside a test file. It is derived from the
router now: every registered route must be in `[benchmark.routes]` as
`measured`, `unmeasured`, or in the reasoned `internal` map. The fast Citadel guard at
`tests/citadel/test_bench_route_coverage.py` fails before build work on a route
in no class, a duplicate classification, an unexplained internal route, or
service/gateway topology drift. `unmeasured` is debt that may only shrink;
`internal` is a one-entry ratchet, not an easy exemption for new routes.

## Benchmarks that are still pytest

Not yet migrated; they remain the gate for their paths.

| what | where |
|---|---|
| VM lifecycle: provision, exec-ready, exec, delete | `tests/capsem-serial/test_lifecycle_benchmark.py` |
| fork, image size, boot-from-image, data survival | same file, `test_fork_benchmark` |
| route latency gate | `tests/ironbank/test_route_latency.py` |
| route latency artifact | `tests/capsem-serial/test_route_latency_benchmark.py` |

Each guards against the latest checked-in evidence by the same config-owned
relative factor. Do not author a second absolute ceiling beside one.

Read [references/guest-benchmarks.md](references/guest-benchmarks.md) before
changing guest categories, storage/rootfs attribution, or protocol deltas.

## When to run

- `just bench-quick` while working on the service or gateway hot paths
- `just bench` before claiming anything about performance
- `just bench criterion` after touching MITM, SSE, JSON-RPC, DNS, the
  interpreters, or the logger's write path
- the lifecycle and fork suites after boot, teardown, image or VirtioFS changes