Xiangdong-415 STEM Experiment Designer
Purpose
Use this skill to design experiments that convincingly test a STEM paper's claims. It focuses on matching claims to evidence, selecting fair baselines, and making evaluation reproducible.
Inputs To Ask For
- Main claims or hypotheses.
- Method or system being evaluated.
- Dataset, sample, simulation, apparatus, or benchmark.
- Constraints such as budget, runtime, safety, ethics, equipment, or available data.
- Target venue and expected standards.
Workflow
Convert claims into testable questions.
- For each claim, define what evidence would support it and what result would weaken it.
Choose baselines.
- Include classical, current strong, domain-standard, and simple sanity-check baselines.
- Explain why each baseline is fair.
Define metrics.
- Use primary metrics aligned with the claim.
- Add secondary metrics for trade-offs: latency, memory, energy, cost, robustness, uncertainty, safety, interpretability, or sample efficiency.
Plan core experiments.
- Main comparison.
- Ablation study.
- Sensitivity analysis.
- Robustness or generalization test.
- Failure-case analysis.
- Runtime or resource analysis when relevant.
Add statistical and reproducibility controls.
- Repeats, random seeds, confidence intervals, effect sizes, significance tests, calibration, train/test separation, blinding, or cross-validation as appropriate.
- Define data splits and exclusion criteria before seeing results when possible.
Map experiments to paper figures and tables.
- Decide which result belongs in the main paper and which belongs in appendix or supplementary material.
Output Format
Provide:
- Claim-to-experiment matrix.
- Baseline and metric rationale.
- Experiment protocol.
- Ablation and sensitivity plan.
- Statistical analysis plan.
- Expected tables and figures.
- Risk list: confounders, weak evidence, missing controls, and likely reviewer objections.
Quality Checklist
- Every major claim has at least one direct experiment.
- Baselines include strong recent competitors and simple controls.
- Metrics match the actual research goal.
- Ablations isolate the contribution rather than merely adding extra numbers.
- Randomness, data leakage, confounding, and reproducibility are addressed.