Expgen

Expgen

Purpose

Expgen is a self-contained experiment-generation skill. It orchestrates preparation, code generation, and rapid validation, then produces a validated exp_handoff.yaml for the downstream exprun skill.

Internal subskills:

• Prepare: read references/exp-prepare.md when grounding the idea, resources, constraints, and milestones.
• Gen: read references/exp-gen.md after prepare passes and code generation should start.
• Validate: read references/exp-validate.md after gen passes and rapid validation is needed.
• Handoff: after validate passes, write exp_handoff.yaml for exprun.

The bundled script scripts/exp_state_guard.py is part of this skill. Resolve it relative to this SKILL.md file, and use an absolute script path when the shell working directory is the user's project. Do not rely on project-local scripts or external repository paths.

Input Sufficiency Gate

Before initializing or advancing the state machine, evaluate whether the current invocation contains enough information for expgen. Use the same gate for full PaperAgent pipeline calls and standalone user calls.

Build an input_context from the current invocation:

• direct user request or experiment request
• research idea, paper claim, hypothesis, or selected idea text
• upstream plan / external memory / plan document, if provided by PaperAgent
• reference papers, code references, resource manifests, dataset descriptions, and explicit output constraints
• requested implementation target, runtime budget, and expected handoff destination when provided

If an upstream plan is provided, compare the current invocation against that plan instead of inventing a separate checklist:

• preserve the plan objective, baselines, datasets, metrics, target modules, milestones, and deliverables
• flag contradictions between the user request and upstream plan
• flag details required by the upstream plan that are missing from this invocation
• override the upstream plan only when the user explicitly changes it or local evidence proves it impossible

If no upstream plan is provided, judge only whether the user-provided information is sufficient. Do not require upstream artifacts just because this is a standalone call.

Minimum information needed to proceed:

• experiment objective and claim to test
• baseline or comparison target
• target method, module, or expected code change
• dataset or data source
• primary metric and comparison protocol
• expected artifact: runnable repo, script, checkpoint, result table, report, or exp_handoff.yaml

If the input is insufficient or contradicts the upstream plan, stop before init / enter prepare and return missing_context_questions. Ask one focused question at a time in a grill-me style, and include the recommended answer when possible. Do not generate experiments from underspecified or contradictory input.

State Machine

Use exp_state.json in the chosen output directory as the single source of truth. Before any stage work, call the guard from this skill package:

python scripts/exp_state_guard.py init --dir <output-dir>
python scripts/exp_state_guard.py enter --dir <output-dir> --stage prepare
python scripts/exp_state_guard.py assert --dir <output-dir> --stage prepare --expect running
python scripts/exp_state_guard.py pass --dir <output-dir> --stage prepare --artifact experiment.md --artifact resource_manifest.yaml --artifact constraints.yaml --artifact milestone_intent.yaml
python scripts/exp_state_guard.py block --dir <output-dir> --stage prepare --blocker "missing dataset"
python scripts/exp_state_guard.py resume --dir <output-dir> --stage prepare --summary "user approved resource change"
python scripts/exp_state_guard.py status --dir <output-dir>

Legal stage order:

prepare -> gen -> validate

Entry gates:

• prepare: may enter after init.
• gen: requires prepare: passed.
• validate: requires prepare: passed and gen: passed.

Hard rules:

• If blocked: true, stop. Do not continue downstream.
• A blocked stage can resume only through resume.
• pass must declare real artifacts; the guard rejects missing artifact paths.
• Repair belongs only inside Validate, and only after prepare and gen have passed.
• Prepare does not auto-download datasets or substitute resources unless the user approves.
• Do not edit exp_state.json by hand except for explicit manual recovery.

The guard also writes events.jsonl and uses .exp_state.lock to prevent concurrent writes.

Multi-Agent Use

Expgen may use subagents for phase-local work when the host agent supports them. The main orchestrator must remain the only actor that calls scripts/exp_state_guard.py.

Suggested subagents:

• Research subagent: inspect experiment.md, local datasets, reference repos, upstream plan context, and prior artifacts; produce method_brief.md.
• Contract subagent: convert the stage goal into bounded acceptance commands, metric schema, interface constraints, and pass/block criteria.
• Gen subagent: implement only the current milestone or stage contract in the editable workspace.
• Eval subagent: run acceptance commands, inspect metrics, review semantic alignment, and recommend pass, repair, or block.

Milestone model:

• Treat each milestone as a subgoal inside the current stage.
• Run each milestone through survey -> contract -> gen -> eval.
• Use subagents only for independent work; merge findings before editing shared files.
• A milestone is complete only when its eval evidence satisfies its contract.
• If the host has no subagent support, perform the same four steps sequentially in the main orchestrator.

Subagent rules:

• Subagents must not edit exp_state.json or events.jsonl.
• Subagents must return artifact paths and evidence, not stage status.
• The orchestrator decides the final state transition after reading subagent outputs.
• If subagents disagree, keep the stage running and create a review note; do not advance by majority vote.
• If evidence is missing, block or repair locally. Do not infer success from agent confidence.

Orchestration

1. Run the Input Sufficiency Gate. If it returns missing_context_questions, stop before state changes.
2. Select or create <output-dir>.
3. If no state exists, run init, then enter prepare.
4. Load the relevant subskill reference for the current stage.
5. Perform the stage work and produce its required artifacts. For generated milestones, use the survey -> contract -> gen -> eval subgoal pattern.
6. End the stage with exactly one guard action: pass, block, or fail.
7. Continue to the next stage only if the guard reports the previous stage as passed.
8. When validate passes, produce exp_handoff.yaml; full execution is handled by exprun.

Evidence Discipline

Borrow these v2-style artifacts without copying its heavy runtime:

• method_brief.md: concise source-grounded interpretation of the experiment.
• contracts/<stage>.yaml: bounded commands, metric schema, and pass/block criteria before implementation.
• reviews/<stage>.yaml: acceptance evidence, semantic check, code/runtime issues, and recommendation.
• snapshots/<stage>/: accepted source/config snapshot for stages that modify code.

These artifacts are advisory evidence. The guard state remains authoritative.

Stage Artifacts

Prepare pass requires:

• experiment.md
• resource_manifest.yaml
• constraints.yaml
• milestone_intent.yaml

Gen pass requires:

• runnable experiment repository
• milestones.yaml
• handoff.yaml
• accepted code snapshot or file manifest

Validate pass requires:

• validate/decision.md
• validate/rapid_validation.json
• repair log if repair occurred
• review evidence that the rapid result still tests the prepared claim

Validate pass also requires:

• exp_handoff.yaml

Blocking Policy

Block instead of repairing when:

• required resource paths are missing;
• metric, baseline, ablation, or claim is undefined;
• generated code changes the experiment meaning;
• validation does not support scaling;
• rapid validation budget or artifact integrity fails;
• the next action requires user approval.

When blocking, record the exact blocker and the smallest user/actionable next step.