Research Experiment Design

Research Experiment Design

Select and test one explicit hypothesis within one RQ with one complete experiment. One experiment may contain an integrated workload, baseline, ablation, and repetition matrix only when every cell tests the same direct relation. Do not require several independent lifecycle questions to all pass for the experiment to be valid; one does not mean one tiny run.

PAPER-VALUE ADMISSION -> PROPOSE -> PLAN REVIEW -> REAL PREFLIGHT -> FULL RUN -> RESULT REVIEW -> RETURN

Do not edit the paper, docs/idea-story.md, or docs/user-instruction.md. Return results and interpretation to the orchestrator for routing and writing. Never change, split, merge, or rename the selected RQ inside this skill.

Boundaries

• Use real systems, agents, models, datasets, traces, or established benchmarks when available. Keep custom code to necessary adapters, instrumentation, and analysis.
• Do not add project-authored experiment-control interfaces: schemas, parsers, promotion gates, repair lineage, exact-budget accounting gates, or equivalents. Use the official or source-native path; otherwise reject it.
• Ordinary reproducibility needs versions, configs, seeds or repetitions when relevant, one runnable command, raw outputs, and an analysis path.
• Plans, reviews, and reports are ordinary Markdown; do not add a second control, integrity, or verdict format. Code, configs, logs, and raw results remain ordinary experiment artifacts.
• Do not invent results or interpret a smoke test as a paper result.
• Treat evidence validity and honest scope as hard constraints. Within those constraints, prefer the experiment that can support the largest credible, consequential, belief-changing paper story, not the easiest defensible local result or unsupported breadth.
• Do not count run IDs, completed reports, checker passes, packaging, or provenance work as research progress or present them as separate scientific contributions; they have value only when necessary to execute or trust an admitted experiment.
• Treat the plan's correctness check, oracle, timeout, and evaluation path as fixed during a run. Changing any of them is a recorded plan deviation that reruns affected comparisons, never a way to make results pass.
• Do not run Git publication actions unless separately authorized.

PAPER-VALUE ADMISSION

Before writing a plan, read the selected RQ, the paper's central thesis and evaluation promise, docs/idea-story.md, docs/evaluation.md, and the current closest-work/baseline map when they exist. Compare the proposed experiment with the existing evidence and the strongest plausible next experiment. Compare live candidates breadth-first before deepening one branch.

Admit the experiment only when the plan can state:

• the largest credible paper story it could unlock without changing the RQ;
• the load-bearing uncertainty or strongest reviewer reject argument it tests;
• the different paper decision produced by a positive, contradictory, mixed, or inconclusive result;
• what independent evidence it adds beyond prior runs and published results;
• why the comparison is not tautological or already settled for the claimed scope, and why it beats the best alternative use of the experiment budget on expected paper-level decision value.

Classify its planned role as headline, decisive, supporting, or dependency. Headline and decisive experiments can change the central story or answer a load-bearing RQ. Supporting work strengthens such an experiment. Dependency work includes smoke, checker, fixture, logistics, packaging, provenance, and setup activity; it has zero independent research value and must normally be folded into the admitted experiment's setup or preflight rather than returned as a paper result.

Reuse existing evidence before running anything new; a new experiment is the last resort. First check whether prior valid results already answer the question — after dropping meaningless baselines, focusing on the relevant subset of the data, reanalyzing raw outputs, or consolidating several existing experiments into one integrated proof. Group existing evidence into the paper's small set of reviewer-facing RQ experiments and admit a new run only when it repairs, falsifies, or materially expands one of them and no reuse or reanalysis path can.

Reject or consolidate a standalone proposal when success and failure lead to the same paper decision; it repeats an established positive control; it merely adds another weak baseline, workload, scale point, checker, or run ID; or its only result is that infrastructure is ready. Return a rejected proposal and the higher-value alternative to the orchestrator; do not execute it to create visible activity.

Admission filters redundant or decision-irrelevant work, not imperfect work. Do not be stricter than the science requires: when the proposal is the strongest runnable next experiment for the RQ, admit it and record its limitations in the plan instead of blocking or demanding an ideal design.

Baselines

One set of rules, applied at admission, planning, plan review, and result interpretation:

• Give every baseline a role: strongest external competitor or current practice, strongest alternative mechanism, component ablation, or lower-bound/null/sanity control. Prefer the strongest runnable fair comparison.
• Treat main-baseline choice as scientific model selection, not a completeness list. A main baseline must represent a credible competing answer to the selected RQ, and matching or beating the proposed method must change the interpretation of that RQ. For each, state the competing position it represents, why existing published evidence is insufficient, and what conclusion follows if it matches or wins.
• Default to citation over rerunning: if a published result already settles the comparison for the claimed scope, cite it instead of running the baseline. Run a baseline only when a matched numerical comparison on this workload is necessary, and then prefer the official implementation or published protocol.
• Label missing-coverage checks, component removals, oracle or upper-bound workflows, and mechanism-engagement checks as controls or ablations, not baselines.
• Keep known-weak, definitionally losing, or already-settled baselines only as minimal rows needed to interpret a stronger comparison. Baseline count is not evaluation strength; do not add weak or redundant conditions merely to increase the comparison count.
• There is no minimum baseline count, and at most 3 main baselines are allowed. The default comparison is the single strongest fair baseline — SOTA or current best practice — plus the controls or ablations needed to interpret it. Add a second or third main baseline only when it represents a different credible competing answer that citation cannot settle; cite or drop every other candidate. Never demand more baselines as a condition of admission or approval when the strongest fair comparison is already present.

PROPOSE

Read the paper's RQ verbatim and write one concise plan using references/plan-template.md. The plan answers:

• What specific uncertainty within this RQ will the experiment resolve?
• What outcome is currently expected, and what competing explanation matters?
• What result would contradict the expectation?
• What published protocol or official artifact can be reused?
• What is the smallest non-redundant set of strongest fair baselines?
• Which workloads and published standard paper-facing primary metrics are necessary, and which paper or official benchmark defines each? Keep project-defined weighting, cutoff-budget scores, and model-reader protocols internal. Which correctness checks, repetitions, and costs are necessary?
• What real command, completion rule, raw-result path, and target figure or table will be used?
• How will the claim-matched primary effect and its uncertainty determine positive, negative, mixed, or inconclusive results, with a secondary metric or control acting as a veto only when it invalidates correctness, comparison fairness, or the exact claim?

Keep all cells that answer the same hypothesis in one comparison matrix. Do not turn each baseline, workload, repetition, or validation cell into an independent paper experiment or research contribution.

State whether a contradictory result would directly challenge a necessary prediction of the paper's central thesis or only bound the tested mechanism, constructor, workload, baseline, or metric. Default to the narrower evidence effect unless the plan directly tests that necessary thesis prediction.

The plan is an agreement about what to run, not an authorization object. If the RQ itself is unclear or scientifically unhelpful, end the protocol and return the problem to the orchestrator as a pending idea question instead of rewriting it here.

If results or plan review force a materially different expected answer, revise the same plan and record the reason in prose. Do this at most twice for the same RQ. A third such change ends this experiment protocol and returns the history and results to the orchestrator; do not start another protocol version. It does not itself trigger idea discussion: routing follows the paper-impact judgment from RESULT REVIEW or another orchestrator trigger.

PLAN REVIEW

Use one fresh reviewer and at most two follow-ups; a new reviewer, name, or tag does not reset the limit. Verdicts and repair ideas are diagnostic, not authority to continue or broaden the experiment. If a blocking defect remains after round three, close the proposal and return the reviewed plan and problem to the orchestrator; do not add more review rounds or infrastructure, and do not rereview an unchanged plan. Keep the discussion in one plan-review.md; reviewers do not edit the plan, and the review must examine each concern below.

• Does the experiment actually test the declared hypothesis and add valid evidence toward the selected RQ, and are the expected and contradictory outcomes meaningful?
• Does the admission rationale hold: load-bearing uncertainty, non-duplicate evidence, best use of the budget, decision-relevant success and failure?
• Are prior work, baselines, workloads, published standard primary metrics, correctness checks, and comparison budgets scientifically fair; does every paper-facing metric have a defining paper or official benchmark citation; and does each baseline satisfy the Baselines rules (credible distinct alternative, genuine need for a matched run, controls/ablations labeled separately, weak or redundant rows removed)?
• Are the real command, data path, repetitions, completion rule, cost, and result interpretation executable and sufficient?

Only a scientific or executability defect that would invalidate the result blocks execution. Optional polish does not, and reviewers must not be harsher than that standard: requests for more baselines, wider workloads, or stylistic preferences are suggestions, never blockers.

REAL PREFLIGHT

Run the smallest real end-to-end case using the actual model or system, real input path, baseline path, metric, and raw-output path. Unit tests, generated fixtures, schema checks, and reviewer simulations are not preflight.

Preflight establishes only that the path runs. Repair a broken runner, data path, metric, or configuration without reopening unrelated plan questions. Close only when the real path cannot run without changing the approved experiment or continued repair is no longer the highest-paper-value action. Do not add a control interface. Permit at most three preflight attempts.

FULL RUN

Use the target repository's documented command, official tool, or installed domain workflow; do not substitute a self-authored toy runner for the authoritative path.

After preflight succeeds:

1. run every planned workload, baseline, repetition, scale, and failure case to terminal status;
2. preserve raw outputs and lightweight checkpoints for long runs;
3. repair systematic execution defects and rerun affected cells;
4. keep negative, failed, excluded, and inconclusive rows with reasons in the repository, never in the paper;
5. when a data or coverage limitation stops an analysis, the limitation itself becomes the next work item: keep repairing coverage (adapters, joins, corpus, harness) and pushing to resolve the question as far as possible, even after genuine attempts; never leave an unanswered question as an escape hatch — if it truly cannot be resolved, state exactly what was tried and why it remains open instead of closing it with N/A;
6. record any necessary deviation and rerun comparisons affected by it;
7. do not interpret a partial prefix as the completed experiment.

Preserve the integrated experiment matrix in one plan and one result report.

Use references/technique-catalog.md only when selecting a systems testing technique; pick the smallest technique that tests the declared hypothesis rather than a venue template bundle.

RESULT REVIEW

Give a fresh reviewer the approved plan, raw-result paths, code/config needed to recompute results, and the selected RQ. Do not provide a desired verdict.

Run one result review per completed run. A finding may invalidate the result, but cannot authorize repair or repeated review of unchanged artifacts.

The reviewer checks completion, commands and deviations, mechanism engagement, metric recomputation, correctness, baseline fairness, uncertainty, leakage, exclusions, negative results, competing explanations, and every proposed paper number or figure. Confirm the metric and correctness check are not defined circularly by the method's own output.

Audit each main baseline independently. A comparison cannot support superiority when the baseline did not engage its intended mechanism, suffered an avoidable interface or implementation failure, used an unfair information or compute budget, or is only a weak proxy for an available official method. Report such a condition as invalid or inconclusive for that comparison rather than counting its failure as a proposed-method win. Keep control and ablation results separate from the headline baseline comparison.

Report separate judgments:

run status: valid / invalid / incomplete
tested hypothesis: supported / contradicted / inconclusive
research value: decisive / supporting / dependency-only / redundant
paper impact: mechanism or workload boundary / additional RQ evidence / direct thesis challenge
next paper decision: <what changes or why nothing changes>

A single experiment is evidence toward its selected RQ, not the answer to the whole RQ. Do not promote a local negative result into a direct thesis challenge merely because the run is valid or surprising. Keep a contradicted result and its error analysis in the repository, then redesign the experiment, algorithm, or system implementation when that is the highest-value path and adds a new causal discriminator. Run a fresh final evaluation after redesign. Never weaken or change the tested hypothesis to fit a result, create indefinite low-value retries, or change the story without explicit user approval.

A valid dependency-only or redundant result does not answer the RQ, enlarge the evidence frontier, or count as research progress. Record it honestly and move to the predeclared decisive work or a higher-value experiment rather than writing it up as an additional result.

Invalid work is repaired only without a prohibited interface and when rerunning remains the highest-paper-value action; otherwise close it. Incomplete work resumes only on the same valid path. Valid admitted results return to the orchestrator; dependency-only and redundant results follow the rule above. Idea discussion triggers only for a direct thesis challenge.

Outputs

Under the active experiment directory, keep one current plan, one file containing all plan-review discussions, raw results, and one result-review report. Update docs/evaluation.md with the selected RQ, tested hypothesis, admission rationale, planned role, plan, run status, raw paths, results, research value, paper impact, next paper decision, interpretation, and remaining uncertainty. Link large artifacts rather than copying them into reports.