Mech Interp Runner

Mech-Interp Runner

This skill drives the frozen bespoke local mech-interp machinery (see archive/experiment/phase1/probe/steering/LEGACY.md). It mints NO new cells. To author a NEW steering / extraction / probe-fit / gate-scoring cell, use the mechinterp-cells skill (tuner-backed mechinterp verbs), not this one.

Use the checked-in scripts and configs to plan, gate, run, and aggregate local mech-interp work. Do not hand-roll terminal loops. This SKILL.md is a progressive-disclosure router: it carries the invariants and routing only. Procedural detail lives in references/, and current results live in docs/sessions/ and experiment-local docs under experiments/<slug>/ - never inline run history here.

Start Here

Always choose the narrowest reference needed for the task:

Do not preload all references. Read only the file needed for the current operation, then follow any further routing inside it.

Quick Commands

Prefer the skill CLI for common local analyses instead of hand-writing long PowerShell commands. It delegates to checked-in repo scripts, resolves paths from the repo root, uses sys.executable, and forces UTF-8 subprocess output (PYTHONUTF8=1, PYTHONIOENCODING=utf-8, PYTHONUNBUFFERED=1) to avoid Windows console encoding failures.

Use --dry-run on any subcommand to print the delegated command before running it. For live Docker/GPU execution the same approval rule applies: do not pass --execute unless the user has approved that live run.

Core Invariants

• The research target is coherent epistemic-humility expression, not a raw refusal axis. A candidate is promising only if it improves one damaged behavior without degrading the paired desired behavior. See references/interpretation-invariants.md.
• Generated-answer replay is the behavioral gate. Offline separability (AUC, d, readout recall) and first-token answer-start movement are screening evidence, never a steering claim.
• Treat outputs as Tier 2 exploratory local mechanism evidence.
• Do not edit synaptic-tuner/.
• Do not run Docker/GPU unless the user explicitly approves that live run.
• Keep base-original h_base adapterless work fail-closed until the live runner explicitly supports adapterless base execution. In DPO/KTO extractions h_base is the SFT-merged pre-adapter model, not original Qwen base.
• Keep generated outputs gitignored by default unless a governed publication decision explicitly whitelists them.
• Legacy reruns keep their registered generation and capture backend. The project-wide vLLM preference applies to new surfaces after the required smokes and bridges; it never authorizes an engine swap inside a frozen cell.
• Stay local to this repository for mech-interp work. Do not use external workflow or memory systems unless the user explicitly asks in the current turn.

Validation

Run focused non-GPU checks after runner/config/skill edits:

python -m pytest experiments/common/knowledge_probe/tests/test_mechinterp_causal_pilot_sweep.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_causal_pilot_runner.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_causal_pilot_dry_run.py -q
python -m pytest experiments/common/knowledge_probe/tests/test_mechinterp_behavior_axis_scan.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_behavior_axis_directions.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_calibrated_expression_plane.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_logit_cell_analysis.py -q
python -m pytest experiments/common/knowledge_probe/tests/test_mechinterp_sae_smoke.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_sae_train.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_sae_feature_analysis.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_sae_feature_directions.py \
  experiments/common/knowledge_probe/tests/test_mechinterp_sae_behavior_feature_analysis.py -q
python -m pytest .skills/mech-interp-runner/tests/test_mechinterp_cli.py -q
python bin/sync_skills.py --check

Compact Recovery Loop

After context compaction or degraded memory:

1. Re-read this skill; drill into references/ only as needed.
2. Re-read the latest relevant docs/sessions/ note and local KG/search results if they affect the next experiment.
3. Set up or run the current local experiment with checked-in scripts/configs.
4. Analyze outputs against the research target (interpretation-invariants).
5. Update session notes, experiment-local runbooks/plans, and this skill when a durable procedure or gotcha changes.
6. Choose the next highest-ROI local slice from the evidence and repeat.

Skill Maintenance

Edit the canonical tree under .skills/mech-interp-runner/ only; .agents/ and .claude/ are generated mirrors. Keep this contract intact:

• Put timeless rules, routing, commands, and validation gates in SKILL.md.
• Put durable procedure and gotchas in one-level references/*.md files and link them from Start Here.
• Put current findings, numeric results, and interpretation snapshots in docs/sessions/*.md and experiment-local docs under experiments/<slug>/ - not in the skill.
• Put reusable analysis logic in checked-in scripts/configs, not prose.
• If a section grows because of one experiment, move the details to a session or experiment note and leave only the general rule plus the reference path.

Before adding a paragraph here, ask: "Will this still guide a future agent six months from now without loading today's run history?" If not, it belongs in a reference or a note. After canonical edits, run:

python bin/sync_skills.py --write --skill mech-interp-runner
python bin/sync_skills.py --check --skill mech-interp-runner