# Benchmark Methodology

> Rules for measuring or quoting any benchmark number in this repo. Load before running the bench tier, writing a delta into a PR or CHANGELOG, or touching anything under benchmarks/.

- Skill: `gabrielmoreira/benchmark-methodology` (Agent Skill)
- Install (CLI): `npx skillmds@latest add gabrielmoreira/benchmark-methodology`
- Raw SKILL.md: https://api.skillmd.com/api/skills/gabrielmoreira/benchmark-methodology/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: gabrielmoreira (https://skillmd.com/u/gabrielmoreira)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/gabrielmoreira/benchmark-methodology

---

# Benchmark methodology

Authority: `docs/harness/ARCHAEOLOGY.md` (the benchmark rows),
`benchmarks/METHODOLOGY.md`, `benchmarks/REPRODUCING.md`, CLAUDE.md
Practice 4, `docs/harness/FINDINGS.md` F-10, F-13, F-17.

Rules, each a pointer, none restated:
1. **Never hand-type a number.** Every figure comes from a run in THIS
   session or the CI-captured artifact `benchmarks/jcm_reference.json`
   (Practice 4). The comparators read that file; there is no estimator.
2. **Per row, never per total** (F-13): the total hid one cause behind
   another. Diff threshold ids one by one.
3. **Five mirrors move together** (Practice 4): `results.md`,
   `METHODOLOGY.md`, README, `provenance/measured.json`, `REPRODUCING.md`;
   `tests/test_provenance.py` is the gate and `--reference` rewrites them.
4. **The reference is captured where the gate runs** (F-13):
   `benchmark.yml` dispatch with `reference=true`; a dev-box number differs
   by CRLF, walk order and a HOME ledger (F-17).
5. **Deterministic configuration** is the bench tier itself: `--offline`,
   pinned corpora (`benchmarks/tasks.json`), the no-network fixture. Add no
   flags.
6. **A refusal is not a zero**: an absent value prints `n/a`.
7. **Floors live only in `harness/thresholds.json`**; read one with
   `uv run python -m harness threshold <id>`.
8. `benchmarks/schema_baseline.json` is the only source for schema-token
   figures (CLAUDE.md "Tier-switch pricing"); the harness counts a
   different payload and the two never agree digit for digit.