Researchstack Experiment Design

Experiment Design

Read ../researchstack/references/workflow.md, ../researchstack/references/venues.md, ../researchstack/references/conference-story-patterns.md, ../researchstack/references/novelty-boundary.md, ../researchstack/references/evidence-contracts.md, and ../researchstack/references/memory.md.

Before proposing new experiments, load relevant memory if it exists:

• known mandatory baselines,
• reviewer risks tied to evaluation,
• hardware or compute constraints,
• past failed experiment branches,
• user preferences about fairness, realism, and reporting.

Design evaluation from claims backward.

For a method paper, also design from mechanism and rival predictions backward. State what the proposed mechanism, the closest method competitor, and at least one rival explanation predict before inspecting the decisive result.

First identify the paper type from conference-story-patterns.md. Match evidence to that type:

• measurement papers need real phenomena, coverage, controls, and external-validity discussion,
• mechanism papers need causal ablations and strong tuned baselines,
• system papers need implementation realism, end-to-end workloads, and operational failure cases,
• algorithm/control papers need action-ranking or decision-quality evidence, not only final reward,
• architecture/co-design papers need cross-layer cost/complexity and scaling/sensitivity,
• artifact/evaluation papers need reproducibility and examples of changed conclusions,
• abstraction papers need the Abstraction Gate as explicit experiments.

Then identify the closest-work boundary from novelty-boundary.md. For each closest competitor, include at least one experiment or baseline configuration that tests whether the claimed gap actually matters. If a strong baseline using one extra heuristic/state variable could close the gap, include that baseline.

For each claim, specify:

• what evidence type is required,
• which baselines are mandatory,
• which metrics matter,
• what ablation isolates the mechanism,
• what stress case could invalidate the claim.

Before building a large custom architecture, include the smallest direct experiment that tests whether a mature established mechanism already solves the problem under the same semantic, resource, and lifecycle constraints.

Produce an experiment matrix with columns:

• claim,
• metric,
• workload or dataset,
• baseline,
• ablation,
• expected outcome,
• rival prediction,
• falsifying or weakening outcome,
• full lifecycle boundary,
• failure interpretation.

Default rigor checks:

• fair baseline tuning and budget disclosure,
• scale sensitivity,
• compute or hardware cost,
• planning, packing, metadata, synchronization, waiting, and output work required by the real deployment lifecycle,
• robustness to parameter changes,
• tail or worst-case behavior if relevant,
• negative results worth reporting.

Flag common paper-killing problems:

• main claim depends on one cherry-picked setup,
• ablations do not isolate causal factors,
• systems speedups ignore resource cost,
• dynamic required work is moved outside the timed region or comparison boundary,
• ML comparisons use stale or weak baselines,
• networking evaluation omits adverse or dynamic conditions.

When the session establishes a durable evaluation rule, baseline policy, or failed direction that should not be relearned next time, propose a project-memory entry for it.