Eval

Eval — design and scaffold benchmarks that can't fool you

Home base: ~/Projects/evals (MACVB + standing bakeoffs + evalcore + scaffold).

Why we run our own (operator framing, 2026-06-13)

Evals here exist to speed up, parallelize, and buy confidence that tokens aren't wasted — not to be stingy. Two corollaries:

• Materiality threshold: a routing/config difference is worth measuring — or even thinking about — at ≥1.5–2× (tokens, wall, or quality). Below that, take the default and move; optimizing sub-1.5× deltas costs more in attention than it returns. (Matches the measured landscape: gated-execution effort tiers differ 0.3–0.65×, i.e. 1.5–3× — material; same-tier rewordings ~1.0–1.04× — noise.)
• Own-benchmark rule: where literature exists, ADOPT its design (Phase 0); where it can't exist — frontier-local, per-release, per-rig properties like dispatch routing, effort knobs, harness/permission behavior, protocol compliance under failure — your own eval is the only instrument there is. Papers will always be a generation behind your rig.
• Ride real work: the cheapest substrate is work that must happen anyway — dispatch the licensed-lane pieces as preregistered arms and the tokens pay twice (evals/dispatch_deletion_edges is the pattern: 3 production deletion edges shipped AND extended the routing table).

Sanity-check tier (below a full eval): a ≤10-call probe with a written 1-line prediction, no scaffold, no DECISIONS row — for "does this lane even work / does this knob even move" questions. Keep the artifact (script + outputs) in the relevant eval dir or /tmp. The moment its result starts steering a default, it must retroactively pass the full gate (see Anti-patterns: "a quick probe that becomes a claim"). Methodology grounding: agent-infra research/2026-06-11-eval-skill-and-evals-repo-improvements.md, research/benchmarking-science-2026.md, research/2026-05-31-eval-benchmark-methodology-delta.md.

An eval is an instrument, not a script. Most eval failures are instrument failures: softball cases that can't discriminate, judges that see candidate names, N too small for the claimed conclusion, golds nobody verified, results with no consumer. Every phase below exists because one of those happened here.

Where does this benchmark live? (phenome / genomics / intel agents read this first)

Two regimes — route by cadence + entanglement + publishability, the same rule BENCHMARKS.md already states. Both already exist in the wild; don't collapse them.

Each repo already has its own eval home — use it; do NOT funnel everything to evals/. The evals repo is specifically for cross-cutting model/engine/tooling-selection that spans repos or is about generic agent infra. Domain and strategy benchmarks stay home:

So: evals/extraction_bakeoff benchmarks intel's and phenome's extract prompts from inside evals/ (cross-cutting model choice), but an intel strategy backtest goes to intel-harness and a genomics variant-accuracy check stays in genomics/benchmarks. The /eval discipline (pre-registration, power, blind judges, invariant-claim split) applies in all of these homes regardless of location.

Tooling — NO symlinks for code. The house pattern for shared Python is a package in substrate/packages/<pkg> consumed via path = "../substrate/packages/<pkg>", editable = true (how corpus-core / corpus-testing reach phenome+genomics). Symlinks are only for data dirs and the AGENTS.md/GEMINI.md→CLAUDE.md doc mirrors.

• evals-repo bakeoffs get evalcore + scaffold + prereg guard for free (evalcore is an editable dep of evals/ — import evalcore.stats / .judge / .leakguard just works).
• evalcore lives in substrate/packages/evalcore (promoted 2026-06-13, ADR 0001 — phenome became the proven 2nd consumer). Pure-stdlib, zero-dep. To use it from ANY repo's in-repo eval, add to that repo's pyproject.toml: "evalcore" in dependencies + under [tool.uv.sources] evalcore = { path = "../substrate/packages/evalcore", editable = true }, then uv sync. Then import:
  • from evalcore.judge import dispatch, assert_blind, lint_not_leading, lint_stakes_neutral, cyclic_assignment
  • from evalcore.leakguard import assert_no_gold_leak — call before EVERY system-under-test dispatch
  • from evalcore.stats import wilson_ci, mcnemar_exact, cohen_kappa, holm_correction, point_biserial, benjamini_hochberg Worked example: phenome/tests/evals/epistemics/judge_refusal.py runs lint_not_leading on its own judge prompt as a standing tripwire (the regression guard for its 2026-06-13 led-judge incident).
• Add it WHEN an eval genuinely needs a primitive, not speculatively — the proven-common bar still holds (vetoed-decisions): a repo whose evals consume none of it should NOT carry the dep. evals/ + phenome consume it today; genomics/intel add the line if/when an in-repo eval needs blind-judge / leak-guard / Wilson-κ / power. Still: NO symlinks, NO copying evalcore's code into a repo.

Phase 0 — Dedup (before designing anything)

cat ~/Projects/evals/BENCHMARKS.md ~/Projects/evals/DECISIONS.md   # settled questions
grep -il "<topic>" ~/Projects/agent-infra/.claude/rules/vetoed-decisions.md

If the question is settled (extraction models, SaC routing, retrieval backend, judge panels...), the answer is the DECISIONS.md row — don't re-run it. New evidence that a verdict is stale → re-open via a new run, never by editing the old verdict.

Also check the external prior art — for the DESIGN, even when the rates are stale. A 2-minute search (Exa/Perplexity for the named bias/benchmark + arXiv) tells you (a) whether a benchmark already exists (don't reinvent — JudgeBiasBench, the SPB PIR/Null-PIR framework, etc. existed and we nearly rebuilt them) and (b) the controls the field already knows you need. "Measure the live model, papers lag the frontier" (frontier-timeliness rule) licenses re-measuring the rate on current models — it does NOT license skipping the literature's method. Papers lag on rates but LEAD on confounds: the controlled design (length-ratio control, truncation control, quality-matched neighborhoods) is scale-independent and transfers. Skipping it is how you repeat a confound the field solved two years ago (see the verbosity egg in Anti-patterns).

For a structured fitness verdict on a candidate external benchmark, dispatch the benchmark-rater agent (global; preloads this skill + model-guide). It grounds the rate-vs-method call in real sources (the paper, the scoring code, training-set membership) and returns USE-AS-IS / ADAPT-DESIGN-ONLY / REJECT plus a contamination–durability sub-verdict (DURABLE / ROTS / NEVER — does it rot as a standing instrument re-run per model release?). Use it before adopting a third-party benchmark and before greenlighting a from-scratch build — its most common output, ADAPT-DESIGN-ONLY with a precise BORROW list, is what stops both a needless rebuild and a naive adoption of a rotting benchmark. Promise-to-release is not released: grade DESIGN from the paper if useful, but RESULT stays non-auditable until the task/grader/record-level ledger and exact run manifest are actually available. A future-tense repository link earns no reproducibility credit.

Phase 1 — Design (the questions that decide if the eval should exist)

Answer in writing (they become PREREGISTRATION.md fields):

1. Construct — what ability/property, operationalized how? One sentence. Estimand before leaderboard: declare whether the unit is a model, a harness, or the deployable model×harness configuration. A configuration leaderboard is valid for exact-system routing; it does not identify a model effect. Model claims need a common harness or a connected factorial grid with model, harness, and interaction effects reported separately.
2. Decision + consumer — which production default / routing rule does the verdict change, and where is that recorded (DECISIONS.md row)? No consumer → don't build. Criterion validity (the NASA-recruiter test): name the real-world outcome the score is a proxy for — the way a selection test is only worth running if it predicts job performance, not test-taking skill (Schmidt-Hunter selection-validity literature). "Extraction F1" matters only insofar as it predicts pipeline usefulness; a benchmark you never check against the downstream outcome is a vanity metric, however internally clean. Where the criterion is measurable, validate against it; where it isn't yet (partial-verifier regime), say so and treat the score as a bounded proxy, not the target. (This is consumption-over-autonomy with a name.)
3. Verifier regime — deterministic ground truth (substring, recall@k, exact answer) or judged? Deterministic PRIMARY decides; judges corroborate. A model-as-judge proxy does not make taste work verifiable (constitution: bad eval is worse than none). Spec↔oracle alignment: a deterministic oracle must enforce exactly what the task states — no more (overly-strict → false-fails correct solutions; a pass then evidences contamination, not skill) and no less (low-coverage → false-passes incomplete ones). Executable ≠ valid; see the mined-task audit under "Grading an EXTERNAL / vendor benchmark". Input availability is part of the prompt: for file/tool agents, the presence or absence of one special file can reveal the intended method. Stage a realistic superset of plausible inputs and run a cue-only baseline. Pre-register the distractor policy and match production clutter; excessive or semantically alien distractors change the construct into triage/context management. For retrieval/ranking: is there exactly ONE relevant item per query, or can the corpus hold co-relevant siblings? One-gold recall@k is valid only in the former; topically-dense corpora need graded multi-doc relevance, or recall@k scores label noise (see Phase 3). For free-text "did they identify X?" grading: substring/anchor matching is paraphrase-brittle and biased against the arm that words it differently (critique_replay: a scored "universal MISS" was a universal HIT — the anchor caught one arm's phrasing). INVERT the default there — a blind dual-family judge (cite-required: a DETECTED with no quotable span is NOT_FOUND) is PRIMARY; substring + cross-arm convergence run as deterministic BACKSTOPS that FLAG (never silently override) disagreements for human resolution (evals ADR 0005). Three judge-transport gotchas that contaminated a real run before its verdict — all caught by reading traces, not buckets:
   • INLINE the payload into the judge prompt; do NOT -f-attach it. Under grading framing the judge silently mis-reads an attached file ("no reviewer findings were provided") though it IS delivered — reproduced deterministically. (Fix belongs in evalcore.judge.dispatch.)
   • A "no findings" / JUDGE_ERROR verdict is TRANSPORT, not a miss — read the rationale; SMOKE one packet before the batch (it caught two transport bugs before any spend; separate transport from capability).
   • Judge competence is ASYMMETRIC — validate each judge against ground truth before trusting a panel. Cross-family-PRIMARY routing assumes equal competence; a weaker judge (gemini-3.5-flash, forced temp=1.0) under-counts the arms it is primary for. κ + human-resolve-disagreements is load-bearing, not optional.
4. Criterion over pipeline — rubric/criterion design explains ~9× more judge-reliability variance than scoring architecture. Spend the hour on the rubric, not on a fancier panel.
5. Contamination plan — per-item provenance: authored-fresh | post-cutoff | public-lifted. Public items contaminate candidates AND judge memory. authored-fresh means the exact text is novel — NOT that the topic is; general-science/medical content is in every model's training data even when your composition isn't, so don't over-claim contamination-freeness.
   • Valid gold for a claim/verification eval is AFFIRMATIVE, never absence. A refuse/abstain/HOLD gold must come from a confirmed contradicting source, retraction, or supersession — NOT from "not in the store / NEI / no evidence found" (closed-world; inverts rankings). Every external claim-verification benchmark's NEI label is absence — do NOT lift it. (Canonical: phenome reference_kb_grounded_eval_defeater_invariant + ADR 0008; the SciFact/JudgeBench/COVID-Fact audits.)
   • The strongest uncontaminated gold is a POST-CUTOFF VERDICT FLIP — harvestable, not just a tag. A claim whose authoritative verdict flipped after the model's cutoff T is contamination-free by construction (the model can only have memorised the pre-flip verdict → it regurgitates the now-wrong belief) AND an affirmative defeater (anchored to a dated authority). That combination is the one thing no borrowed benchmark supplies. Harvest LIVE (recall forbidden — past your cutoff too) from dated-authority feeds filtered to change_date > T: Retraction Watch, the FDA DSC table, ClinVar delta, the CPIC cpic-data git log. Cite the primary dated record, not a blog's crawl date (mirrors re-surface old posts with fresh dates). This is intel-harness point-in-time with T = the training cutoff. Method + first harvest: ~/Projects/evals/docs/post-cutoff-flips/.
   • Enforce the flip-gold with the SEAL, not trust — and split judgment from retrieval. Flip-gold is clean only if the SUT can't search its way to the post-T answer mid-eval, so pair it with allow_internet=false (the DeepSWE seal below): the airgap turns "we assume it didn't cheat" into "it provably couldn't." This also dissolves the "but it doesn't test web-browsing" objection by SCOPE: a judgment-over-a-staged-claim eval (adjudicate promote/hold/reject given an evidence packet) is packetizable + airgappable; retrieval ("find the right materials") is a SEPARATE capability where PIT-constraining a LIVE search is unsolved (URLs mutate in place at stable addresses, date metadata lies, engines have no as-of-T mode — intel-harness's unsolved web-PIT problem). Claim-verification ESCAPES that trap precisely because a claim+evidence bundle is a static object you curate as-of-T once; trading can't, its task is integrating an unbounded flowing stream. Curating the packet enforces the PIT boundary by construction — don't try to PIT a live search. If you must test retrieval, do it against a FROZEN snapshot (Wayback/Common Crawl at T) and state the coverage ceiling; don't pretend live fetch is as-of-T.
   • Same-family confound — at EVERY model touchpoint (generator / judge / reranker-fusion): any model that helps produce or score the eval's relevance signal must share no lab/family with any candidate. ≥2 neutral-family generators and judges (blinding does NOT protect — the bias rides the content of the grades); a candidate-family reranker informs, never gates. Measured incidents + mechanics: references/anti-pattern-evidence.md §Phase-1.
   • Cheap pre-screen before building an embedder bakeoff at all: embed the corpus with both candidates, run a handful of queries through the production stack, measure top-10 overlap. >~0.85 ⇒ the swap is immaterial — don't build the full graded eval. (v2 measured 0.89: through hybrid+rerank the two embedders returned ~9 of the same 10 docs. The full eval confirmed HOLD, but the overlap screen predicts "switching changes ~1 in 10" in minutes — run it at the §0 gate.)
   • Frontier judge bias (measured 2026-06): position/order bias is SOLVED on frontier judges — stop spending order-swaps/position-debias there. Same-lab judges share a style signature (a same-lab panel ≈ 1 effective vote; Bradley-Terry SE on it is false precision) → use cross-lab panels, read disagreement as signal. Our "verbosity bias" probe was RETRACTED (length-ratio artifact). Rates, arXiv refs, re-run controls: references/anti-pattern-evidence.md §Phase-1.
   • Judge NOISE BUDGET — a single-trial judged number is PRELIMINARY (distinct from bias): ~13.6% mean single-trial flip rate at frontier; for a decision repeat the judging (≥3) or PPI/PRECISE-correct; a same-lab multi-judge panel is NOT independent votes. Numbers + refs: references/anti-pattern-evidence.md §Phase-1.
6. Invariant ambition — what mechanism-level claim could this eval produce that survives a config swap? If only a local verdict is possible, fine — say so up front.

Phase 2 — Scaffold

cd ~/Projects/evals && just new-eval <slug>
just power <N>            # paste output into PREREGISTRATION.md; declare SCREENING|CONFIRMATORY
git add <slug>/PREREGISTRATION.md && git commit   # prereg FIRST — the guard enforces this

Golds are platinum: mechanically checkable or human-verified. Never audit golds by asking an LLM to re-check them (auditors re-solve and trust themselves: wrong-reference detection 68%→9% at scale).

Phase 3 — Discrimination probe (before budget)

uv run python3 <slug>/run.py --probe on ≤10 items. Required: a trivial-pass baseline score AND one case that separates candidates. Both flat → fix cases, not N. (Evidence: File-Search-vs-emb "parity" was title-matching softballs; discriminating rerun: 8/10 vs 4/10.)

The probe is NECESSARY, NOT SUFFICIENT — eyeball misses + check gold validity. A low trivial-baseline rules out lexical leakage; it does not rule out label noise or over-obfuscation. Before trusting ANY recall/nDCG/accuracy number:

• Eyeball ≥10 actual misses — read the query, the gold, and what scored above the gold. If the higher-ranked items are plausibly co-relevant/correct, your label is wrong: you are scoring label noise as model failure (and can penalize the BETTER system, which clusters co-relevant items). Cheapest validity check there is; the most-skipped. Watching aggregate ranks is NOT this — look at what beat the gold.
• Single-gold → graded relevance on topically-dense corpora. One-gold recall@k is valid only if exactly one item is relevant per query. If the corpus has sibling/overlapping items (multiple docs on a topic, near-dupes), collect a relevance set (LLM-judge the top-k union + human spot-check), score graded nDCG/MAP. A verdict that flips across analyst choices (generator, pool, window) is the tell that label noise ≫ signal.
• Don't over-obfuscate to beat the baseline. Aggressive keyword-stripping can push queries into unrealistic riddles that test abstraction, not retrieval — match the real query distribution. (Evidence: bio_embedding_bakeoff 2026-06-11 — kw-baseline 0/12 looked rigorous, but 23/48 queries had the gold outranked by co-relevant sibling memos; the single-gold recall@5 verdict was inconclusive, surfaced only by eyeballing misses post-hoc — audit_label_noise.py.)

This generalizes beyond retrieval — read the TRACES, not the buckets, for ANY eval, and ESPECIALLY on a clean/perfect score. Aggregate buckets are a proxy; a verdict on unread traces verifies the arithmetic, not the construct. Persist traces by default (a harness that discards them is unauditable). For judged refusal/decline/classification evals: read what the judge actually wrote (is its prompt leading?), and read what the opposite-label control did (a refusal eval with no ENDORSE foil cannot tell "refuses the bad" from "hedges on everything"). A perfect score INCREASES the obligation to read. (Evidence: phenome KG-verifier 2026-06-13 — a committed "16/16 decision-grade" verdict had unread, in fact unpersisted, traces; the trace audit found a led judge + no specificity control.)

Phase 4 — Run + stats (evalcore does the discipline)

• Judges via evalcore.judge.dispatch(..., blind_to=[all candidate names]) — blinding is enforced (raises), stakes-framing linted, temperature pinned. One strong judge + one diverse-family κ instrument; majority-of-panel is NOT truth (~2 effective votes). cyclic_assignment when judges × scenarios ≥ 2×2. Schema includes confidence.
• Rows via evalcore.results.row/append_rows; provenance via provenance() (prompt hashes).
• Report per-stratum, never only global. Paired comparisons: paired_bootstrap_diff, mcnemar_exact + holm_correction; prob_superiority_beta is the small-N primary readout.
• SCREENING declaration → lead with ranks + effect sizes + CIs; p-values secondary.

Statistical canon (Miller "Error Bars to Evals" + Biderman "Lessons from the Trenches", full-text in agent-infra research/2026-06-14-eval-methodology-canon.md) — the parts not already enforced above:

• SE + n on every decision-grade number. CLT sqrt(Var/n); Bernoulli sqrt(p(1-p)/n) ONLY for strict 0/1 scores — it's WRONG (too wide) on F1/partial-credit/judge scores (the Llama-3 report shipped this error). A comparison without an SE is not decision-grade.
• Cluster the SE when items are grouped (shared passage, one prompt × N langs/paraphrases, multi-turn on one scenario) — up to 3.05× wider on real data; an unclustered grouped CI lies about precision.
• ≥2 seeds/temps; report mean + variance; NO single-run headline (Miller + Biderman + BetterBench: 14/24 benchmarks fail this). Cut variance by resample K=4–10 or next-token-probs — never by lowering temperature (that changes what you measure).
• Separate format-compliance from correctness: log RAW and extracted output; a regex extractor can 0-score a correct answer. Pin the harness (exact prompts + extraction code + model-version + commit) — a score without it isn't reproducible. Any parse/format rerun is part of the treatment: predeclare a cap, preserve the first attempt, charge every retry to tokens/time/cost, report per-arm retry counts, and never silently remove unparseable outputs from the intended correctness denominator.

Don't trust vendor leaderboards as rankings (research/2026-06-14-benchmark-leaderboard-methodology-critique.md): LMArena's Bradley-Terry is sound but the board is structurally captured (Leaderboard Illusion — Meta tested 27 private variants; data-access = +112%); Artificial Analysis is a SOLID screen but 33% of weight flows through LLM judges (use per-axis, not the composite); CursorBench/OpenRouter measure cherry-pick/spend, not capability; SWE-bench Verified ≫ original. Steal the contamination-resistant DESIGNS (temporal holdout > canary > fuzzy-dedup), not the leaderboards.

Phase 4.5 — Trace audit BEFORE the verdict (mandatory gate)

Recurred 2× operator-forced (phenome KG-verifier 2026-06-13; Cursor Composer 2026-06-14 — both times a committed verdict was overturned only after "go look at the traces"). This is now a GATE, not advice: no DECISIONS row, no committed verdict, until this audit is written. A verdict from aggregates you didn't trace-check is a draft, not a result.

Hook-enforced (evals + phenome + genomics + intel, deployed 2026-06-14): pretool-eval-preflight.sh BLOCKS any eval-runner (run*/judge*/score/dispatch-arm) that sits in an eval-design dir (an EXPERIMENT.md/PREREGISTRATION.md marker in its dir/parent, OR a path under an evals?//benchmarks?//tests/evals/ segment) until the agent records a <evaldir>/.preflight-ack confirming the design checklist + these trace-audit pre-commitments. One confirm per eval; fails open.

Mechanize checks 1–2 — run the item analyzer (don't eyeball the matrix). Psychometric item analysis catches the outlier/mis-keyed item that a human scanning a table misses. Emit your response matrix as long-format JSONL ({"model","item","score","scale_max"}, one row per cell) and:

# evalcore evals: ZERO hand-emit — trials.jsonl IS the matrix (variant=model, item_id=item, scores)
uv run python3 ~/Projects/skills/eval/scripts/item_analysis.py --adapter evalcore <run_id>.trials.jsonl
# non-evalcore: emit long-format yourself  (or --adapter phenome|intel)
uv run python3 ~/Projects/skills/eval/scripts/item_analysis.py matrix.jsonl

It computes per-item difficulty + discrimination (corrected item-total r) + top-model dispersion and prints a ranked INSPECT list. Every flagged item must be trace-audited before the verdict — they are leads, not conclusions (at small N it says so). What the flags mean:

• INSPECT-GOLD (negative discrimination — the best models score worst) → the gold is likely mis-keyed/contaminated. This is the diekstra signature: composer's 0/33 was correct, the gold was wrong. The analyzer flags it #1 mechanically (validated 2026-06-14); you no longer have to be told "go look at the traces."
• CEILING/FLOOR (difficulty →1 / →0) → ~zero information; prune or replace (a saturated item carries no signal — the "injected-defect benchmark" failure mode).
• TOP-DISPERSION (high-ability models disagree) → ambiguous gold or a real capability split.

Normalize per scale_max (a 0–3 faithfulness scale is NOT a 0–1 recall scale — the analyzer's own first bug). The analyzer is the front-end; the five checks below are the judgment it can't make:

Adversarial cross-model complement (the spirit-audit). item_analysis is mechanical on the matrix; for the traces, run the integrity lens via a different-lineage model — it independently catches contaminated gold, broken-arm-scored-as-result, confounds, and saturation (validated 2026-06-14: Composer, fed the traces blind, caught diekstra + corroborated the saturation finding). ~/Projects/skills/analyze/scripts/spirit_audit.sh <PREREGISTRATION.md|EXPERIMENT.md> <trace_files>… fans Composer over the artifacts; or /critique with the composer axis. Lens: analyze/lenses/spirit-audit.md.

Run these five checks; record the result in EXPERIMENT.md §5 (or a *_RESULTS.md § Spot-check):

1. Outliers first. For each arm, list per-item scores, not just the mean. Any item far from the arm's others (e.g. recall 96/82/0) → READ THAT TRACE before averaging it in. A mean over a bimodal/outlier distribution is a lie ("52% mid-pack" was really 96%/82% + a correct []). Tiny/near-empty outputs (a 1KB output among 30KB ones) are red flags, not data points. (The item analyzer above ranks these for you; this check is reading the traces it points at.)
2. Is the GOLD/grader valid on contested items? A model scoring 0 may be doing the RIGHT thing against a gold that violates its own contract. Verify: does the gold honor the task's own drop/keep rules? (Composer scored 0/33 by correctly dropping methodology claims the contract excludes — the gold + every other model extracted them = contract violation. The 0 was correct.) If the gold is wrong for a case, the metric is INVALID there — say so; don't rank on it. Pair with the judge's GOLD_INVALID escape (Anti-patterns).
3. Inter-judge / inter-rater agreement. Before reporting any judged number as ground truth, check whether judges agree on it. If they split materially (opus said 24 unsupported, gpt said 45), report the RANGE and flag the arm as hard-to-judge — don't launder one judge's count into a verdict.
4. Every arm appears in the OUTPUT. Hardcoded model/candidate lists silently drop a newly-added arm even when its raw files parse fine. Confirm the new arm is in every summary table/row, not just on disk.
5. Attribution honesty. If the conditions you're contrasting differ in ≥2 ways (contract × domain × prompt-length), you CANNOT attribute the effect to one of them — name the confounds (cross-ref the confound anti-pattern). State the mechanism evidence you DO have, separately from the correlation.

Cheap rule of thumb: read ≥1 trace per arm and every outlier trace. The cost is minutes; the cost of a committed wrong verdict is a re-audit + a correction commit + lost trust (measured twice).

Phase 5 — Verdict + invariant extraction

1. EXPERIMENT.md §6a local verdict (config-bound) → DECISIONS.md row + the production change (or explicit no-change). The eval is done only when that row lands.
2. EXPERIMENT.md §6b invariant claims — mechanism, invariant-to, transfer evidence or UNTESTED-TRANSFER. These rows are the publishable residue; a bakeoff with zero invariant rows is still useful, just not publishable. (Metrology precedent: in 2019 the kilogram was redefined from a drifting physical artifact — Le Grand K — to a fixed Planck constant, because a standard must be a durable INVARIANT not a perishable artifact. §6a local verdicts ARE the artifact that drifts; §6b mechanism claims are the invariant you keep.)
3. Deviations from prereg → §7 Limitations, explained, never silently absorbed.

2026-06 frontier adopts (digest)

Cross-axis convergence of 5 frontier memos: outcome-only scoring is structurally insufficient — verify the trace/structure. One line per adopt below; the mechanics, arXiv evidence, and the DeepSWE / LifeSciBench confirmations live in references/frontier-adopts.md — read it before implementing any of these.

• Isomorphic verifier — for agentic/tool-use SUTs, score the trace (the path that reaches the verdict), not just the output.
• Gold-leak guard — evalcore.leakguard.assert_no_gold_leak(sut_prompt, gold) before EVERY SUT dispatch; hold out ≥30% of scoring dimensions.
• Block mirror domains — retrieval/web SUTs block huggingface.co / paperswithcode.com / benchmark mirrors (search-time contamination ~3–4% of queries).
• Rubric decomposition before grading — decomposed judges κ≈0.79 vs ≈0.51 holistic; never a holistic LLM judge as primary.
• Item-quality flag + FDR — evalcore.stats.point_biserial flags suspect items (investigation flag, never auto-gate); benjamini_hochberg is EXPLORATORY-only.
• Read ≥5 traces — ≥1 pass + ≥1 fail per top arm against the 4 questions (mirror-read / loophole / dangerous action / scaffold limit); cite verifiable trace anchors.
• Metamorphic vocabulary — state §6b invariants as source_relation ⇒ output_relation; the oracle-free invariance tier is discovery-only.
• Q-matrix tags — gold-only 3–5-dim capability ontology per case; do NOT fit IRT at N=10–60.
• Deterministic-grader constructs (LatchBio) — per-item separation table, sentinel gold fields, before-step snapshot gold, method-name suppression, reproduce-or-discard gold, anti-hint input supersets, cost+trajectory as first-class axes.
• Measurement-science canon — anchor-equate a drifting SUT (NEAT), SDT d′-vs-criterion split (critique_replay/sdt.py), proper scoring rules for confidence outputs, RoB-2-style graded pre-flight.
• Peer prediction (gold-free scoring) — evalcore.elicit + anchor jury; validated ρ=1.0 vs gold ordering; FAILS under ≥50% colluding pool (enforced in code, never assumed).
• Capability is the LAST-RESORT hypothesis — a score is capability evidence only after contamination/shortcut/memorization are affirmatively excluded (named hard gate).
• Verdict = machine-checkable gate-ledger — evals ADR 0007 + scripts/check_verdict.py; confidence/call is EARNED by discharged gates; a pass must cite a leaf.
• Guards — PPI/CLT-PPI invalid below 50 labels/stratum (just power refuses); fitted IRT / CapBencher / CAT / noise-injection sandbagging stay deferred (evals/docs/decisions/deferred-and-open.md).
• Process reflex (extends Pre-Build #1) — before inventing a metric or grading scheme, inventory the measurement sciences for the existing instrument.
• Capability CURVE over test-time compute (AISI 2026-07-02) — a fixed-budget agentic score is a LOWER BOUND, not a point: newer models gain disproportionately from budget, so fixed budgets understate frontier gaps most exactly where you care. Sweep the budget, report the curve, label serial vs parallel allocation; a curve still rising at the cap ⇒ report "≥score @ cap", never a ceiling. (The per-arm token logging that eval-token-costs already mandates IS the curve's x-axis.)
• Exhaustive/completeness constructs — three adopts (Harvey LAB: Diligence, evals/research/2026-07-19-lab-diligence-rating.md) — for any eval where "found 95% but missed one material item = wrong" (needle-in-haystack recall, deal-killer provisions, exhaustive enumeration): (1) all-pass co-primary + diagnostic pair — report all_pass (every item found) beside the criterion/recall rate; the gap between "almost complete" and "complete" is the signal, and all-pass is the construct-honest headline (but a stricter bar — at hard-task N it may not discriminate, so keep the graded rate as the small-N decider, cf. wandr_exhaustive §A2.1). (2) scoped per-criterion judge context — an LLM judge for item i sees ONLY the output file(s) that item declares relevant, not the full deliverable — measurably cuts cross-item verdict contamination (extends "rubric decomposition"; N/A to deterministic graders). (3) source-traceable planted gold — each gold item carries a pointer to the specific document/source it was planted in (LAB's sources field), making gold auditable to a source rather than to judge inference — the deterministic-grader analog is a per-member re-derivation query (wandr_exhaustive §A2.2).
• Verified-code / formal-proof constructs — three adopts (Lanyon linear-benchmarking teardown, evals/research/2026-07-22-lanyon-linear-benchmarking-rating.md) — for any eval scoring code + a machine-checked proof/verification artifact (Lean/Coq/SMT or equivalents): (1) escape-hatch mechanical scan — grep the proof artifact for the prover's bypass primitives (sorry/admit/axiom/native_decide) as a DETERMINISTIC reward-hacking backstop under any judged verification scoring; a direct instantiation of the isomorphic-verifier adopt + ADR-0005 deterministic-backstop pattern we already held (hindsight: HAD-PARTS). (2) proof-attachment verdict ladder — grade whether the proof attaches to the EXECUTED artifact: faithful / partial (honest but confined to dead code or a degenerate regime) / misformalized (rests on escape hatches or theorems that don't establish what they appear to) / disconnected (proves a different object than what ships) — sharper spec↔oracle-alignment vocabulary than bare pass/fail. (3) terse-prompt degenerate-substitution probe — pair each detailed prompt with a terse variant and check whether the SUT silently narrows the problem to a trivial special case before "succeeding" (Lanyon caught a CFL=1.0 exact-shift swap-in for a requested general second-order scheme) — the letter-of-the-ask trivialization failure mode, cheap to add to any agentic eval. (The write-up's own RANKING is discard-tier: judge pool = candidate pool — all five SUTs peer-grade each other with no independent judge — and zero released transcripts/harness.)

Independently confirmed by DeepSWE (withheld grader; seal-the-env-or-domain-block; the free-executable-oracle boundary — don't cargo-cult behavioral verifiers into oracle-free domains) and LifeSciBench (strong construct ≠ transferable ranking; grader-named-or-fail; validation-printed-or-unvalidated). Full teardowns in the reference file.

Review mode

Given an existing eval dir: check each phase artifact exists and bites — prereg committed before results (git log --follow), power declaration matches N, probe evidence present, misses eyeballed (what outranks the gold, not just aggregate ranks) + single-gold validity on dense corpora, judge payloads blind (grep for candidate names in judge prompts/payload builders), per-stratum tables, DECISIONS.md row, §6b filled or explicitly empty. Report gaps as a table; fix all confirmed gaps, not top-N.

Grading an EXTERNAL / vendor benchmark (the benchmark-rater agent inherits these — it loads this skill; LifeSciBench teardown 2026-06-18, evals/research/2026-06-18-lifescibench-rating.md):

• Grade DESIGN and RESULT separately — they earn different verdicts. A provider self-eval's construct + rubric can be genuinely borrow-worthy while its RANKING is non-transferable (the launch model tops a self-graded board). ADAPT-DESIGN-ONLY is the common right answer; never carry a vendor ranking to a DECISIONS row until an independent cross-lab grader reproduces it.
• Distinguish MODEL-vendor from SERVICE-vendor COI — the second is diffuser but still real. The canonical self-eval is "lab fields a model in the pool AND grades it" (OpenAI-grades-OpenAI). A service/infra vendor that ships no model but benchmarks the exact capability it SELLS is a different, weaker-but-present shape: the benchmark doubles as a sales instrument and (often) a customer training-data pipeline. Mitigants that actually move the verdict: independent third-party REPRODUCTION on a private held-out set (not just a re-graded public sample), unflattering headline results (frontier failure, churn at #1 across labs), open + independently κ-audited grading. Absent all three for a given SLICE (e.g. a brand-new extension), that slice's numbers stay uncorroborated regardless of the parent benchmark's standing. (Harvey LAB: Diligence — service-vendor shape, base LAB well-reproduced, diligence slice 0 days of independent reproduction; evals/research/2026-07-19-lab-diligence-rating.md Q6.)
• Grader-named-or-fail. If a model-graded benchmark does not NAME the grader model in the paper, treat judge family-neutrality as FAILED by default — you cannot rule out a same-family judge. (LifeSciBench's grader is GPT-5.5, same family as the winning GPT-Rosalind, and surfaces only in a press quote; the paper says merely "model-assisted grading, where used.")
• Validation-printed-or-unvalidated. A grader-validation study that is DESCRIBED but whose human-agreement numbers (κ / correlation / MAE) are not PRINTED counts as unvalidated — don't credit it. (LifeSciBench §5.3 promises "we report correlation, MAE, pass/fail agreement"; no value appears in the paper.) "Expert-authored" via an anonymous/withheld contributor pool (the DRACO pattern) is likewise face validity you cannot audit.
• Mined-tasks ⇒ task-validity sample-audit before adoption (spec↔oracle alignment). Tasks mined programmatically from issue/PR/

…(truncated)