Research Experiment Plan

Research Experiment Plan

Quick start

1. Freeze the claim, decision rule, and anti-claims before listing runs.
2. Classify the intended evidence as exploratory, confirmatory, independently verified, or operational/high-stakes; use ../research-pipeline-planner/references/epistemic-assurance-contract.md when promoting beyond exploratory status.
3. Decide whether this is a standalone plan or the experiment stage inside an orchestrated suite.
4. Initialize the full experiment pack with scripts/init_experiment_pack.py whenever the plan needs tracked execution.
5. Build the minimum decisive experiment blocks, not a benchmark wishlist or a paper-defence script.
6. Run a cheap non-vacuity preflight before expensive or confirmatory execution.
7. Separate must-run from nice-to-have runs, attach explicit decision gates, and emit bridge-ready outputs for later execution.
8. For orchestrated execution, declare inputs, evaluator artifacts, required outputs, allowed lineage relations, and gate-conditioned downstream activation using ../research-pipeline-planner/references/experiment-execution-binding.md.

Constants

• MAX_PRIMARY_CLAIMS = 2 unless multiple inseparable claims are truly required.
• MAX_CORE_BLOCKS = 5 unless the evaluation setting forces more.
• MAX_BASELINE_FAMILIES = 3 unless the field norm demands additional families.
• DEFAULT_SEEDS = 3 when stochastic variance matters and budget allows.

Modes

Standalone mode

• Work from the user prompt plus any local notes or result files already present.
• Do not require a suite root.
• If the user asks for project-level sequencing, current-state inspection, or coordination across multiple research stages, invoke research-pipeline-planner first instead of treating experiment planning as the whole task.
• If the user asks what idea to pursue, or the claim is still only a broad research direction, invoke research-idea-discovery before experiment planning.
• Collaboration is still allowed: if novelty review, results audit, or review-loop artifacts exist, use them; if one of those skills would materially improve the plan, recommend or invoke it.

Orchestrated mode

• Prefer the canonical directory ./experiment-plan/.
• Read upstream artifacts from research-brief.md, artifact-index.md, ./ideation/, ./novelty-review/, and ./literature-review/ when present.
• Keep the experiment outputs easy for downstream paper planning and review to consume.
• Carry forward selection history, evidence class, negative evidence, and material predecessor failures rather than resetting them at the experiment stage.
• Treat claim-map.json and run-blocks.json as frozen inputs when creating an experiment-bound harness work item.
• Keep technical episode completion, scientific disposition, and decision-gate result distinct. A correctly executed negative experiment may complete while failing its scientific gate.

Input contract

• Minimum:
  • a concrete claim or hypothesis
  • an evaluation goal or target decision
• Prefer:
  • baselines under consideration
  • constraints on data, compute, or deadlines
  • known reviewer objections
  • existing novelty or literature artifacts
  • current and requested evidence class
  • case-selection history
  • material predecessor failures and prior negative results

Hard stops

• Stop if the claim is still too vague to falsify.
• If no concrete claim exists yet, route to research-idea-discovery instead of inventing one inside the experiment plan.
• Stop if the evaluation target is undefined.
• Stop if the proposed experiment block mixes too many interventions to interpret cleanly.
• In standalone mode, do not force extra structure when a compact direct answer is sufficient.
• Do not design a confirmatory route around cases selected because they already yield the desired answer; preserve them as exploratory and define a separate selection rule.
• Stop confirmatory promotion when the decision or loss contract omits a decision-relevant error, omission, skip, null, retry, or failure state.
• Stop confirmatory promotion when the evaluated system can read hidden truth, the oracle and runner share undisclosed hard-coded policy, or the claimed independence is only a role label.
• Do not treat structural validation, internal agreement, self-hashes, copied digests, environment flags, or field presence as proof that the claimed assurance property holds.
• Do not treat a technically completed run as a passing scientific gate.
• Do not use experiment lineage to encode a D3-D4 paper pivot; route that through the research commitment contract.
• A failed non-vacuity preflight normally forces revision or a weaker evidence class; it does not prohibit exploratory work.

Output contract

• Primary files:
  • experiment-plan.md
  • experiment-tracker.md
  • claim-map.json
  • run-blocks.json
  • decision-gates.md
  • execution-bridge.md
• In orchestrated mode, these live under ./experiment-plan/.
• In standalone mode, any target directory is valid.
• claim-map.json is the machine-readable source of truth for claims, anti-claims, evidence class, decision rules, loss contracts, falsification tests, and predecessor failures.
• run-blocks.json is the machine-readable source of truth for experiment blocks, dependencies, decision-gate links, pass/fail criteria, selection rules, non-vacuity checks, outcome accounting, hidden-information controls, execution declarations, and allowed lineage relations.
• decision-gates.md records the checkpoints that can halt, narrow, reclassify, or authorize later blocks after a result is interpreted.
• execution-bridge.md translates the plan into implementation-ready instructions without forcing another skill to reverse-engineer the planning intent.

Workflow

0) Load the proposal context before inventing runs

• Read the strongest available upstream artifacts first:
  • research-brief.md
  • novelty-review/novelty-report.md
  • literature-review/*.review.md
  • review-loop/REVIEW_STATE.json
  • existing result tables or audit notes
• Extract:
  • problem anchor
  • dominant contribution
  • optional supporting contribution
  • reviewer-relevant failure modes
  • data, compute, and deadline constraints
  • current evidence class and outcome-informed selection history
  • material predecessor failures and unresolved negative evidence
• If these are missing, derive the same fields explicitly from the user prompt before planning any block.

1) Freeze the claim map

• Use references/claim-evidence-map-template.md.
• Write:
  • primary claim
  • optional supporting claim
  • anti-claims to rule out
  • minimum convincing evidence
  • current and requested evidence class
  • complete decision rule and loss or outcome contract
  • falsification test
  • material predecessor failures and their disposition
• Cap the number of primary/supporting claims aggressively.
• Define what result would force reframing, reclassification, or abandonment.
• If the claim map is unstable, revise it before planning runs.
• Record the claim map in both experiment-plan.md and claim-map.json.

2) Build the experimental storyline before the detailed blocks

• Start from a compact default storyline and delete any block that does not test a decision-relevant claim:
  • main anchor result
  • novelty isolation
  • simplicity or elegance check
  • frontier-necessity check when a frontier-model-era component is central
  • failure analysis or qualitative diagnosis
• Do not optimize only for a favorable paper story. A block exists to discriminate among claims, actions, or explanations, including outcomes that weaken the project.
• Mark each storyline block as:
  • main paper
  • appendix
  • cut
• A stronger modern baseline is preferable to many weak baselines.
• If the project is intentionally non-frontier, say so explicitly and skip the frontier-necessity block rather than forcing one.

3) Run the non-vacuity preflight

Before expensive or confirmatory runs, check that:

• at least one plausible case makes competing systems, policies, or actions differ;
• every decision-relevant error, including failure to act, is penalized by the loss or outcome contract;
• the comparator can win under a plausible condition rather than being disabled by construction;
• case selection was not conditioned on the oracle answer or desired outcome;
• skipped, failed, null, and retried cases remain visible in the accounting.

Record the result in decision-gates.md and in each affected block's non_vacuity_check. If the preflight fails, revise the plan or keep the evidence exploratory.

4) Build decisive experiment blocks

• Use references/experiment-plan-template.md.
• Use references/experiment-block-schema.md.
• Group runs into named blocks with a single purpose each.
• Every block must answer a reviewer-relevant question, not just produce another number.
• Label each block:
  • must-run
  • nice-to-have
  • defer
• For every kept block, specify:
  • claim tested
  • anti-claim ruled out
  • why this block exists
  • dataset / split / task
  • compared systems
  • decisive metrics
  • setup details
  • success criterion
  • failure interpretation
  • expected paper artifact
  • compute budget as a planning note, not a repository-enforced assurance property
  • dependencies
  • decision gate ID
  • case-selection rule
  • non-vacuity check
  • complete outcome accounting
  • hidden-information controls
  • actual independence requirements
  • execution mode and entrypoint when known
  • declared input snapshot paths
  • declared evaluator snapshot paths
  • required output paths
  • allowed lineage relations
• Write the block objects to run-blocks.json rather than leaving the critical structure only in prose.

5) Tighten controls and ablations

• Use references/control-and-ablation-checklist.md.
• Require a fair comparison protocol and the minimum ablations needed to isolate the claimed factor.
• Flag hidden changes such as altered data, training time, search budget, or model capacity.
• A simplicity check should usually compare the final method against an overbuilt or tempting extra-component variant.
• A frontier-necessity check should compare the chosen modern component against the strongest simpler plausible alternative.
• Treat agreement among implementations that share policy, code, data, or hidden truth as correlated evidence until diversity is demonstrated.
• State whether review is self-review or materially independent across context, data, implementation, evaluation, and advancement authority.

6) Build the run order and decision gates

• Use references/run-order-template.md.
• Use references/decision-gates-template.md.
• Put must-run blocks first.
• Add stop/go gates so later runs depend on what earlier runs actually show.
• Every must-run block needs:
  • a gate bound to that block;
  • a condition that advances the plan;
  • a condition that forces revision or weaker evidence classification;
  • a condition that stops the plan.
• A dependent block that requires a scientific outcome must declare a gate-conditioned activation rule in the harness, not only a structural dependency.
• Track expected outputs and lifecycle state in experiment-tracker.md.
• Use tracker statuses:
  • planned
  • ready
  • blocked
  • running
  • analyzed
  • decisive
  • inconclusive
  • dropped
• Carry material predecessor failures forward until new evidence resolves them; reclassification, replacement, or omission is not resolution.

7) Emit the execution bridge

• Use references/execution-bridge-template.md.
• For each must-run block, record:
  • exact upstream claim IDs
  • decision gate ID
  • required inputs and datasets
  • declared input and evaluator snapshot paths
  • expected command or implementation entrypoint if known
  • required output artifacts the auditor or paper planner should look for
  • intended lineage relation and parent-run requirement
  • blockers that must be resolved before someone starts coding or submitting jobs
  • evidence class and whether outcome inspection would trigger reclassification
  • hidden information unavailable to the evaluated system
  • all failure, skip, null, and retry states that must be retained
  • idempotency and restart requirements
  • any downstream activation rule governed by the gate
• Keep execution-bridge.md concise and implementation-facing. It exists so later stages do not have to reconstruct planning intent from a narrative plan.

8) Record risks and collaboration hooks

• Use references/risk-confound-checklist.md.
• If novelty is still uncertain, pull in research-novelty-review.
• If existing results already exist, pull in research-results-auditor.
• If the plan will later feed a draft or response to reviewers, keep outputs legible to research-paper-plan and research-review-loop.
• Validate tracked packs with scripts/validate_experiment_pack.py before treating them as stable stage artifacts.
• The default validator profile remains structural for backward compatibility. For confirmatory or high-stakes promotion, run it with --assurance-profile confirmatory; even that checks field presence, links, and internal consistency, not whether the declared controls actually held.

References

• references/experiment-plan-template.md
• references/claim-evidence-map-template.md
• references/experiment-block-schema.md
• references/run-order-template.md
• references/decision-gates-template.md
• references/execution-bridge-template.md
• references/control-and-ablation-checklist.md
• references/risk-confound-checklist.md
• references/tabmol-ddi-ood-adapter.md
• ../research-pipeline-planner/references/experiment-execution-binding.md
• ../research-pipeline-planner/references/epistemic-assurance-contract.md

Scripts

• scripts/init_experiment_pack.py: create the full experiment-planning pack in a standalone directory or the suite's experiment-plan/ directory, including execution and lineage scaffolds.
• scripts/validate_experiment_pack.py: validate required headings, JSON structure, reciprocal claim/block links, gate bindings, execution declarations, lineage policy, tracker states, and confirmatory assurance fields without claiming executor isolation or scientific validity.