Experiment Runner

Locked Matrix Experiment Runner

Use the checked-in runner scripts to expand, gate, stage, and inspect locked matrix cells. This skill is orchestration glue: it talks to the synaptic-tuner submodule only through materialized recipe YAML and public tuner CLI verbs. It must not add experiment-specific code or config to the tuner.

Amendment A / v0.4 is signed as a prospective extension (user approval, 2026-06-14). Sequential SFT -> DPO and SFT -> KTO arms are not part of the locked v0.3 matrix, are not present in config/matrix.yaml, and must be materialized/run only as deliberate Amendment A cells with separate recipes and run records.

Start Here

Always choose the narrowest reference needed for the task:

Do not preload all references. Read only the files needed for the user's current operation, then follow any further routing inside that reference.

Quick Commands

Core Invariants

• The matrix SSOT is config/matrix.yaml; per-arm default recipes live under archive/experiment/phase1/recipes/; provenance records live under archive/experiment/phase1/run_records/.
• run_matrix.py must assert the pre-registered counts: 19 @ 4B, 9 @ 8B, and 2 bridge cells. Never loosen these assertions to absorb a matrix edit.
• Launch/cancel/delete actions require exact user approval in the current conversation, especially cost-incurring cloud actions.
• One amendment/experiment = one branch off an up-to-date origin/main in its OWN git worktree (.worktrees/<branch>) = one PR. NEVER swap branches in the primary working tree — live GPU queues and monitors run scripts from it in place. Do the full arc in that worktree (recipes, run records, scored results, doc verdict), then PR into protected main. Amendments proceed in PARALLEL, each in its own worktree; never stack a second amendment on another amendment's branch or worktree, and never push to main directly. See reference/operator-discipline.md.
• The no-pollution rule is sacrosanct: runner code may not import tuner internals, write committed files under synaptic-tuner/, or register Epistemic-specific tuner behavior. Ephemeral staging under already-gitignored tuner scratch space is the only allowed tuner-tree write.
• Do not guess tuner CLI flags. Check synaptic-tuner/tuner/cli/parser.py or python tuner.py --help before relying on a flag.
• Both lanes are safety-gated by live capability probes for per-cell seed and beta forwarding. If the probe fails, skip/flag the cell rather than working around the gap in the runner.
• Treat bounded local diagnostics and Amendment A/B evidence as non-headline unless a protocol/run record explicitly says otherwise. Headline numbers come only from the pre-registered default cells.
• Pick the right instrument before writing one: signed protocol revision (headline surface / claims) vs Amendment (a new exploratory evidence cell, falsifier pre-stated) vs lab notebook (smoke/preflight/diagnostic/re-run/authorized-knob tuning). Reserve Amendments for new evidence cells; route lighter work to the session note + run record. Every amendment pre-states a prediction, a falsifier, and its gates — and never moves the goalposts after the result. See reference/amendment-vs-lab-notebook.md.
• Abstention grading: an exact-phrase refusal detector alone is not an acceptance instrument. Any harness whose gates score abstention pairs a frozen, pinned detector (screen) with a pre-registered BLINDED adjudication lane covering detector-negative rows from BOTH populations (benefit and cost), with manifest-before-grading and hash-before-unblinding enforced in code, and a falsifier that closes the regress (no rescoring lane behind the lane). Protocol, rubric, cautionary case, and pre-sign checklist: reference/abstention-grading.md.
• Cost/harm gate diagnosticity: a cost gate whose denominator is an unconditional population (not filtered to rows the intervention actually fired on) measures baseline properties, not intervention cost, whenever the fired fraction is small — and is vacuous by construction at a fired fraction of exactly zero. Counterintuitively, such a gate gains power as the readout gate's separation gets WORSE, and any Wilson-upper-CI cap has a computable minimum-N floor below which it cannot pass regardless of performance. A locked/registered gate's PASS still stands as registered; the diagnosticity caveat travels forward with the result rather than reopening the verdict. Before trusting any cost gate result, decompose its denominator into fired vs unfired and report the fired fraction alongside the verdict. See reference/gate-diagnosticity.md.
• Data-exhaust at build time: every generation harness persists per-sample generation text and the full sub-grade dict in its gitignored row-level run log; booleans-only logs are a build defect, and the pre-sign review checks the persistence schema. Rationale and the H3 cautionary case live in the data-exhaust skill's "Build-time requirement" section. Containment is unchanged: text never leaves gitignored analysis/. This is structurally enforced: open the row-level run log through experiments/common/runlog_contract.py's open_generation_runlog, and declare the manifest's text_capture field (see the experiments skill's Manifest schema).
• Backend choice is part of the evidence surface. New unsteered generation prefers pinned vLLM with batch invariance; new full-depth extraction prefers vLLM only after the model-specific HF bridge passes. Parity-locked cells keep their registered backend. Read reference/batched-generation.md before signing any GPU cell that generates or captures hidden states.
• GPU smoke/preflight before every full generation run (PI standing directive 2026-07-16): before any full steering/generation run, execute a small-N preflight on the SAME code path as the full run — a few rows per family at the dose extremes (bottom rung, reference setpoint, top rungs), verifying per-row readback against the commanded setpoint under the amendment's registered tolerance and recording observed well-formedness at the extremes as a collapse-location estimate. The full-run entrypoint REFUSES to start unless the preflight wrote its pass marker (enforced in code, not by convention), and live first-batch plus per-rung-completion assertions hard abort mid-run on any readback violation. A preflight FAIL is a gate event for the lead, never an operational retry: there is no retry-until-pass (unlike CG1's explicit VOID_REGRADE_ONCE, readback has no registered retry remedy), and any tolerance or rung change it motivates is a signed-config amendment (repin, recorded reason, PI approval) made BEFORE generation. Write tolerances with the instrument's physics in mind: bf16 readback carries a roughly fixed ABSOLUTE error floor (~0.001-0.005 dose_abs observed on Qwen3.5-4B hs20 / Mistral-7B hs16), so a purely RELATIVE tolerance is unattainable at small absolute setpoints — prefer "rel <= X OR abs <= X * reference_dose" so the gate keeps wiring-defect detection power (defects like gain-squared exceed both bounds by >10x) without tripping on quantization dust. Cautionary case: margin-mapping M1 (2026-07-17), whose preflight caught exactly this pre-run; NOTEBOOK entries there record the full adjudication pattern. For how a multi-stage GPU pipeline should be launched and chained around gates like this one, see the mechinterp-cells launch discipline: ../mechinterp-cells/reference/modal-launch.md.

Matrix At A Glance

See reference/matrix-expansion.md for the full mapping and count-assertion contract.

Skill Maintenance

Edit the canonical tree under .skills/experiment-runner/ only. .agents/ and .claude/ are generated mirrors. After canonical edits, run:

python3 bin/sync_skills.py --write --skill experiment-runner
python3 bin/sync_skills.py --check --skill experiment-runner

When this skill grows, move details into one-level files under reference/ and link them from Start Here; keep SKILL.md as a progressive-disclosure router.