Experiments

Experiments Lifecycle

The repo keeps one self-contained directory per evidence-producing experiment under a top-level experiments/ tree. Any experiment type belongs here: a steering cell, a training run, an eval, a probe-fit, or a lab diagnostic. This skill is the tooling around that layout: it scaffolds a new experiment, pins its instrument at signing, tracks its status, and regenerates the human and machine registries from the manifests. It never launches or scores an experiment; that is the job of the type-specific runner skills.

Start Here

Layout

experiments/
  REGISTRY.md          # GENERATED human table (never hand-edit)
  registry.json        # GENERATED machine dump (never hand-edit)
  <slug>/              # one experiment; dir name == manifest slug
    experiment.yaml    # thin machine-readable manifest (SSOT for state)
    AMENDMENT.md       # signed prose: motivation, design, prediction, falsifier, gates, outcome
    NOTEBOOK.md        # running lab log
    cell.yaml          # instrument config(s), pinned at signing
    gates.yaml         # pre-stated pass/fail thresholds (when applicable)
    .gitignore         # ignores directions/ and analysis/
    analysis/          # untracked local scratch (not committed)
    directions/        # gitignored fitted-direction data
  common/              # artifacts promoted for use by >1 experiment (see promotion rule)

Prose lives in AMENDMENT.md; machine state lives in experiment.yaml. Never duplicate the prose into the manifest. The registry files are generated from the manifests and are the only files you must not edit by hand.

Always create this skeleton with bin/exp new --title "<title>" --type <t> rather than hand-authoring the files. The command creates the directory, manifest, AMENDMENT.md, NOTEBOOK.md, placeholder cell.yaml and gates.yaml, and the local .gitignore template in one pass. The preferred multiplayer bootstrap is:

bin/exp new --title "<Experiment Title>" --type <t>

The CLI derives the slug from the title, creates experiments/<slug>/, and stores the title in the manifest. You may still pass an explicit slug when the slug needs to differ from the title. The slug is the durable experiment ID; do not reserve or encode a global amendment letter in the slug for new work. If a legacy letter must be displayed for a migrated record, keep it as compatibility prose/metadata, not the canonical ID.

A teaching or example artifact sets registered: false in its manifest. It still validates structurally but is excluded from claim requirements (it does not need a prediction, falsifier, or verdict) and should not be read as evidence. It still appears in the generated registry, marked teaching artifact:, so the inventory stays complete.

experiments/common/ is a reserved directory, not an experiment: it is the shared cross-experiment code home (graders/, renders/, and promoted directions/). It carries no manifest and is excluded from validation, the manifest scan, and the registry.

Manifest schema

experiment.yaml is the single source of truth for machine-readable state:

slug: <dir name>                 # must equal the directory name
title: <human title>             # filled by `exp new`
type: steer-cell | training-run | eval | probe-fit | lab-diagnostic | historical-amendment
status: draft | signed | running | resolved | null-result | falsified | historical
registered: true                 # false = teaching/example, excluded from claims
created_at: "YYYY-MM-DDTHH:MM:SSZ"  # filled by `exp new`
question: <one sentence>
prediction: <one sentence>       # required to sign
falsifier: <one sentence>        # required to sign
text_capture: enabled | not-applicable | "textless: <reason>"  # see below; scaffolded as `enabled`
checkpoint: {repo: ..., revision: ...}   # optional
instrument:
  configs: [cell.yaml, gates.yaml]       # instrument files pinned at signing
  modules: []                    # optional grader/render/harness modules, pinned too
  pins: {}                       # relpath -> sha256, filled by `exp sign`
  repins: []                     # append-only audit trail, filled by `exp repin`
  persistence: {}                # relpath (from modules) -> persistence declaration
inputs: []                       # repository paths or typed local artifact declarations
pr: <int>                        # optional, the PR that carries this experiment
verdict: <one sentence>          # filled at resolve
kg: []                           # typed KG node ids, filled at/after resolve

status, pins, and verdict are managed by the CLI; do not hand-edit them. historical-amendment / historical is reserved for imported legacy governed records whose original amendment prose is the provenance source; do not use it for new experiments.

Portable inputs and machine readiness

An input that every clone must contain stays a repo-relative string. Portable validation treats it as a repository dependency and fails if it is absent:

inputs:
  - experiments/common/readouts/probe.py

A gitignored or externally staged artifact uses a typed mapping:

inputs:
  - path: scratch/checkpoints/sft-seed1/final_model
    availability: local
    source: locked training-regimen SFT seed-1 run
    sha256: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef  # optional

availability is repository or local. A local input must include source with enough provenance to identify its producer. sha256 is optional for legacy artifacts; when present, it is the file digest or deterministic tree digest checked by bin/exp doctor. Paths must be repo-relative and cannot use ...

bin/exp validate is portable. It checks the declaration on every machine but does not require local bytes to exist. It still requires repository inputs. Before any experiment consumes its inputs, run bin/exp doctor <slug> on that machine. Doctor fails on every missing input and on every declared digest mismatch. This separates commit hygiene from run readiness without weakening the run gate.

Persistence declarations (kill-resume safety)

A signed CPU or GPU module that buffers results in memory and writes output only at the end loses the entire run if it is killed one minute before finishing. instrument.persistence closes this gap at the tooling level: it is a mapping, keyed by the same module relpath used in instrument.modules (same shape as instrument.pins), where every module declares up front how it survives a kill. Each entry is one of:

instrument:
  modules: [harness.py, pool_builder.py]
  persistence:
    harness.py:
      persistence: incremental
      checkpoint_path: experiments/<slug>/analysis/runlog/harness.jsonl
    pool_builder.py:
      persistence: short-run
      measured_smoke_wall_clock_s: 42.5

• persistence: incremental with a checkpoint_path (repo-relative or analysis/-relative): the module writes per-item results through a resumable run log as it goes (see experiments/common/README-runlog.md for the RunLog import path and log-path convention) rather than only at the end.
• persistence: short-run with measured_smoke_wall_clock_s (a number): the module's own smoke run was timed and finishes comfortably inside the window where losing the whole run to a kill is an acceptable risk. A process whose projected wall-clock exceeds about 15 minutes must not use this mode; see the mechinterp-cells SKILL.md persistence invariant.

Config-only instruments (instrument.modules: [], everything routed through generic tuner verbs) have nothing to declare. The sole hard-enforcement point is bin/exp sign, which REFUSES to sign an experiment whose instrument.modules contains an entry with no matching instrument.persistence key, or whose declaration is malformed (wrong persistence value, missing checkpoint_path on incremental, or a non-numeric measured_smoke_wall_clock_s on short-run). sign is the mandatory gate every new experiment already passes before it can run, so enforcement is intact going forward without needing a second gate elsewhere. bin/exp validate only ever prints a WARNING (non-blocking, at every status including draft) for the same gap: this keeps validate from retroactively failing on a stale draft that predates this field, or one that was run informally without ever going through exp sign (a Tier-2 exploratory cell, say), while still surfacing the gap for anyone reading validate's output. Before signing an incremental module, run the kill-resume smoke drill described in the mechinterp-cells reference/organization.md "Kill-resume smoke drill" section: a validator or grep check that RunLog is imported is not evidence that resume actually works.

Before signing ANY cell, verify the registered real-run entry point actually exists and reaches the real path, not only that the smoke passes. A smoke suite exercises component functions; it structurally cannot detect that the orchestration branch of main() is a stub or a refusal placeholder, because the smoke takes its own branch. A cell in this program was signed and merged with every component function smoke-covered while main() contained only the --smoke branch and a hard-coded refusal; the defect surfaced only at execution time. The sign-off check is: run the registered entry point in its real mode (a --dry-run that resolves every real input and prints the execution plan without computing is the standard shape; every new orchestrator module should ship one), or at minimum read main() end to end and confirm a branch performs the registered stages. Library-only modules (no if __name__ guard, no CLI) count as unreachable unless a pinned driver module demonstrably invokes them for real data; check that the driver exists before signing, not after.

Text capture (data-exhaust build-time rule)

text_capture declares whether this experiment's generation harness, if any, persists per-row generation text -- the data-exhaust build-time requirement (.skills/data-exhaust/SKILL.md) made structural. exp new scaffolds enabled; a generation-bearing harness should open its row-level run log through experiments/common/runlog_contract.py's open_generation_runlog, which enforces this at write time. Set not-applicable for an experiment with no generation step at all (pure analysis, probe-fit over existing extractions). Set textless: <non-empty reason> only when generation happens but text capture is deliberately disabled; the same reason must be passed to open_generation_runlog's textless_reason, which folds it into the run log's own meta fingerprint. bin/exp validate requires this field (hard error, not a warning) for any experiment whose created_at is on or after the requirement's effective date; earlier experiments are exempt.

Lifecycle

draft ──sign──> signed ──run──> running ──resolve──> resolved | null-result | falsified
historical  # imported legacy record; not a launchable lifecycle state

1. draft (exp new): scaffold the directory and fill created_at. Fill question, prediction, falsifier, the instrument configs, and write the AMENDMENT.md design. Nothing is pinned yet.
2. signed (exp sign): the instrument is frozen. exp sign computes the sha256 of every file in instrument.configs (and any listed modules), records them in instrument.pins, and flips the status to signed. From here on, exp validate fails if a pinned file changes, so the goalposts cannot drift silently. Signing refuses if prediction or falsifier is empty.
3. running: set by hand when the run is launched (the runner skills own the launch). Pins are still enforced.
4. resolved / null-result / falsified (exp resolve): stamp the one-sentence verdict and the terminal status. exp resolve prints a kg-ingest checklist; ingest the result as typed KG nodes and record their ids in kg:.

Instrument repair (repin)

exp repin <slug> <relpath> [<relpath>...] --reason "..." is the one sanctioned way to change a pinned instrument file after signing. It is legitimate ONLY for a build-environment or harness-crash repair on a signed experiment BEFORE any run artifact exists: for example, a dependency conflict discovered when the Modal image first builds, or a harness bug that stops the cell from launching at all. It is never a way to change the design, and never legitimate once results exist: a repin after resolution is goalpost movement.

repin re-hashes the named file(s), updates instrument.pins, and appends an audit entry per file (file, old_sha256, new_sha256, date, reason) to the append-only instrument.repins list. The reason lands in that audit trail, so the repair is on the record. It hard-refuses everything that would be dishonest: a draft (nothing is pinned yet; edit freely and sign), a resolved/terminal experiment (results exist), a file that is not already pinned, a file whose bytes have not actually changed (a no-op repin), and any repin attempted while an UNRELATED pinned file has drifted (fix the intended file only and investigate the rest). exp validate accepts the repins field and additionally checks that the last repin entry per file agrees with the live pin, while still failing on any pin drift exactly as before.

bin/exp repin <slug> cell.yaml --reason "Modal image dependency conflict fix (pre-launch)"

Generated indices

experiments/REGISTRY.md (human table) and experiments/registry.json (full machine dump) are GENERATED from the manifests by exp regen, sorted by slug and free of timestamps so they are byte-stable. Both carry a "GENERATED - do not edit" header. Never hand-edit them: change a manifest, then run bin/exp regen and stage the result. The reserved experiments/common/ directory is skipped, and registered: false rows are rendered with a teaching artifact: marker, so the registry stays a complete inventory without presenting teaching artifacts as claims.

The .githooks/pre-commit hook enforces this. When experiments/ exists it runs portable exp validate and exp regen --check; a stale registry fails the commit with an instruction to run bin/exp regen and stage the output. Local artifact availability belongs to bin/exp doctor <slug> before use, not the commit hook. Install the hooks once with git config core.hooksPath .githooks, or run a single commit through them with git -c core.hooksPath=.githooks commit.

Promotion rule for shared inputs

An experiment's own artifacts stay inside its directory. The first time a SECOND experiment needs to consume an artifact produced by another, promote that artifact to experiments/common/ and point both consumers at the promoted copy via their inputs: list. The promoted copy keeps provenance: record where it came from (the origin experiment slug and the path it was generated at) in a short note beside it. This keeps cross-experiment dependencies explicit and prevents an experiment from reaching into a sibling's private directory.

Terminology annotations on signed docs (semantic renames)

When a program-wide terminology ruling (recorded in papers/common/terminology.md) retires a term that appears in an already signed experiment's working label or prose, the signed text is NEVER rewritten and nothing is ever renamed on disk. The rename is semantic only and is recorded additively:

1. Precondition: the ruling exists in papers/common/terminology.md first. That file is the sole source of truth for how retired terms render in prose; do not invent a rendering inside one experiment.
2. Append a dated ## Terminology annotation (additive, YYYY-MM-DD) section to the experiment's AMENDMENT.md, stating: which working label predates which ruling, the operational prose rendering, and the sentence "Nothing else changes: the slug, directory, filenames, config keys, gate definitions, question, prediction, falsifier, and every registered constant remain verbatim as signed. This annotation is semantic only and moves no goalpost."
3. Add a matching dated NOTEBOOK.md entry recording who directed it and that no pinned file changed (AMENDMENT.md is not sha-pinned, so no repin is involved; bin/exp validate must stay OK).
4. Never touch the question, prediction, falsifier, or title fields in experiment.yaml, and never edit the signed Question / Prediction / Falsifier / Gates prose. If a rename cannot be expressed without touching those, it is not a semantic change and requires a signed revision with changelog and PI approval instead.
5. Slugs, directories, filenames, and config keys are provenance and stay verbatim forever, in citations too; only running prose renders the new term.
6. Lead executes with explicit PI direction; the change reaches main through a normal PR the PI merges (for a running cell, it may ride the results PR and must be noted in the NOTEBOOK when added).

Command reference

All commands run through the bin/exp wrapper (Windows: bin\exp.cmd), which executes the mirror under .agents/skills/experiments/scripts/exp.py.

type is one of steer-cell, training-run, eval, probe-fit, lab-diagnostic, or migration-only historical-amendment. bin/exp sign reminds you, when a pinned config carries a tuner surface: block, to set surface.expected_config_sha to that config's pin so the tuner aborts on drift.

Skill Maintenance

Edit the canonical tree under .skills/experiments/ only. .agents/ and .claude/ are generated mirrors. After canonical edits, run:

python3 bin/sync_skills.py --write --skill experiments
python3 bin/sync_skills.py --check --skill experiments

Tests live under .skills/experiments/tests/ and run with python3 -m pytest .skills/experiments/tests.