design-experiment
Purpose
Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness are modes.
When to use / not applicable
Use after a hypothesis has an operational construct and discriminating prediction. Not applicable when the hypothesis or measurable outcome is absent.
Input contract
mode_contracts:
factorial: &experiment_input
required: [falsifiable_hypothesis, outcome, factors, constraints]
optional: [baseline, candidate_models, budget, robustness_axes]
constraints: [analysis_plan_must_be_preregistered]
ablation: *experiment_input
comparison: *experiment_input
scaling: *experiment_input
robustness: *experiment_input
Execution protocol
Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.
- You MUST load skill
identify-variables to operationalize the outcome, factors, controls, and their functional roles. You MUST load skill enumerate-dimension-values to define admissible levels or perturbation values.
If the resource envelope or feasibility constraints cannot support an executable design, consider analyze-constraints-readiness before committing to a mode.
- Choose
factorial, ablation, comparison, scaling, or robustness mode.
- You MUST load skill
specify-metrics to preregister metrics, estimands, directionality, and decision thresholds. You MUST load skill estimate-sample-size to derive the sample or repetition requirement. You MUST load skill select-statistical-method to select the inference or estimation method before observing outcomes; specify power and the stopping rule.
If the proposed metric or validator may share artifacts, labels, or assumptions with the system under test, consider audit-validator-independence before freezing the analysis plan.
- You MUST load skill
construct-design-matrix to construct the runnable matrix for the selected mode. You MUST load skill design-randomness-protocol to define seeds, repetitions, and propagation rules. You MUST load skill specify-execution-environment to capture interpretation-relevant hardware, software, data, configuration, and versions. You MUST load skill specify-reproducibility-protocol to define and test the intended reproduction level. You MUST load skill optimize-design-under-budget to select a feasible information-efficient design under the declared resource envelope; include resource and failure checks.
Once the design has produced observations and the task changes from planning to inference, analyze-experiment-results may be the better next tactic.
Mode branches
For comparison or robustness, You MUST load skill select-experimental-baseline to choose a controlled baseline matched to the claim.
factorial: vary multiple factors in a structured design so main effects and interactions are estimable within the declared resource envelope.
ablation: remove or replace components systematically to attribute the outcome to individual parts and suspected interactions. You MUST load skill map-ablation-components to define ablatable units, dependencies, and legal removal or replacement operations.
comparison: evaluate a target against controlled baselines with matched confounds, compute, tuning effort, and preregistered tests.
scaling: instantiate geometric or otherwise justified scale points to test how the outcome changes across the declared regime.
robustness: perturb relevant conditions or inputs and measure whether the claimed effect survives the defined stress space.
Output contract
mode_contracts:
factorial:
produces: [factor_level_matrix, factor_level_catalog, estimands_main_effects_interactions, metric_significance_plan, sample_power_plan]
delta_fields: [findings, decisions, uncertainties, open_questions]
ablation:
produces: [ablation_matrix, baseline_anchors_full_minimal, attribution_contrasts, component_interaction_plan]
delta_fields: [findings, decisions, uncertainties, open_questions]
comparison:
produces: [controlled_baseline_comparison, matched_confound_controls, seed_environment_protocol, statistical_comparison_plan, reproducibility_protocol]
delta_fields: [findings, decisions, uncertainties, open_questions]
scaling:
produces: [scaling_axes, geometric_scale_points, scaling_experiment_grid, curve_fit_plan, scale_budget_plan]
delta_fields: [findings, decisions, uncertainties, open_questions]
robustness:
produces: [perturbation_stress_matrix, severity_axes, baseline_comparison, degradation_metrics, survival_criteria]
delta_fields: [findings, decisions, uncertainties, open_questions]
Thresholds and quality gates
experiment-design HARD-GATE and Budget Gate remain mandatory; no exit before declared minimum yield.
- Resource and run-scale gates are relative to the declared resource envelope and eligible factor/condition space; record numerator, denominator, batch increment, stopping reason, and source references.
- Scale, ablation, comparison, and robustness coverage must reach a justified relative floor over the declared design space; do not substitute an unreasoned fixed run count.
- Factor levels, comparison baseline, statistical test, significance threshold, sample-size rationale, and stopping rule must all be explicit.
- Significance threshold must be pre-registered, not chosen post-hoc.
- Budget-constrained design must report at least one feasible design under the stated resource envelope.
- Scaling mode retains the source criterion of a geometric progression, typically 4-8 points, while evaluating coverage relative to the declared scale domain. Relative audit: declared universe = eligible scale domain; numerator = scale points instantiated; batch increment = points added per design pass; stopping reason = predeclared coverage or saturation rule; source references = scaling-design ledger and design records; direction/threshold rationale = geometric spacing preserves regime sensitivity, with 4-8 retained as the source band.
Failure and counterexamples
Reject post-hoc factor selection, outcome leakage, missing control, unpowered comparison, or a design whose claimed inference exceeds its measured outcome.
Provenance map
experiment-design, factor-level-design, ablation-design, comparison-design, scaling-design, robustness-design, statistical-method-selection, reproducibility-protocol, budget-constrained-design; mode-specific steps retained, agent dispatch removed.
Legacy context checkpoint / Delta notes
Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.
Preserved source criteria ledger
| source |
source line |
kind |
source criterion |
| experiment-design |
45 |
textual |
## HARD-GATE |
| experiment-design |
47 |
textual |
Before entering this campaign, the following must be satisfied: |
| experiment-design |
64 |
textual |
4. How to ensure reproducibility |
| experiment-design |
79 |
textual |
## Budget Gate |
| experiment-design |
83 |
numeric |
\ |
| experiment-design |
84 |
numeric |
\ |
| experiment-design |
85 |
numeric |
\ |
| experiment-design |
86 |
numeric |
\ |
| experiment-design |
95 |
textual |
## Minimum Yield |
| experiment-design |
97 |
textual |
Every campaign invocation must produce at minimum: |
| ablation-design |
43 |
textual |
## Budget Gate |
| ablation-design |
47 |
numeric |
\ |
| ablation-design |
48 |
numeric |
\ |
| ablation-design |
49 |
numeric |
\ |
| ablation-design |
50 |
numeric |
\ |
| ablation-design |
51 |
numeric |
\ |
| comparison-design |
34 |
textual |
- Fair Comparison Protocol (Bouthillier 2021): Control all confounds, same compute budget, same tuning effort. |
| comparison-design |
47 |
textual |
6. reproducibility-protocol (tactic) -> Ensure all results are reproducible |
| comparison-design |
50 |
textual |
## Budget Gate |
| comparison-design |
54 |
numeric-table |
\ |
| comparison-design |
55 |
numeric |
\ |
| comparison-design |
56 |
numeric |
\ |
| comparison-design |
57 |
numeric-table |
\ |
| scaling-design |
43 |
numeric |
2. level-specification -> Define scale points (geometric progression, typically 4-8 points) |
| scaling-design |
49 |
textual |
## Budget Gate |
| scaling-design |
53 |
numeric |
\ |
| scaling-design |
54 |
numeric |
\ |
| scaling-design |
55 |
numeric |
\ |
| scaling-design |
56 |
numeric |
\ |
| robustness-design |
50 |
textual |
## Budget Gate |
| robustness-design |
54 |
numeric |
\ |
| robustness-design |
55 |
numeric |
\ |
| robustness-design |
56 |
numeric |
\ |
| robustness-design |
57 |
numeric |
\ |
| robustness-design |
58 |
numeric-table |
\ |
| statistical-method-selection |
30 |
numeric-table |
\ |
| statistical-method-selection |
31 |
numeric-table |
\ |
| statistical-method-selection |
32 |
numeric-table |
\ |
| statistical-method-selection |
33 |
numeric-table |
\ |
| statistical-method-selection |
34 |
numeric-table |
\ |
| statistical-method-selection |
36 |
numeric-table |
\ |
| statistical-method-selection |
48 |
textual |
- Is the significance threshold pre-registered (not chosen post-hoc)? |
| reproducibility-protocol |
17 |
textual |
# Tactic: Reproducibility Protocol |
| reproducibility-protocol |
21 |
textual |
1. Assess Reproducibility Requirements -> Determine level needed (exact, statistical, conceptual) |
| reproducibility-protocol |
22 |
textual |
2. seed-protocol-design -> Design random seed strategy for all stochastic components |
| reproducibility-protocol |
24 |
textual |
4. Define Verification Plan -> How to confirm reproducibility (re-run subset, cross-machine test) |
| reproducibility-protocol |
25 |
textual |
5. Document Non-Determinism -> Identify and document unavoidable sources of variance |
| reproducibility-protocol |
29 |
textual |
\ |
| reproducibility-protocol |
37 |
textual |
- Are all random seeds documented and controllable? |
| reproducibility-protocol |
38 |
textual |
- Is the full software environment captured (versions, dependencies)? |
| reproducibility-protocol |
39 |
textual |
- Are hardware-specific non-determinisms identified (GPU atomics, cuDNN)? |
| reproducibility-protocol |
40 |
textual |
- Is there a verification protocol (re-run N times, check variance)? |
| reproducibility-protocol |
44 |
textual |
- Is there a plan for cross-machine reproducibility testing? |
| budget-constrained-design |
34 |
numeric |
\ |
| budget-constrained-design |
35 |
numeric |
\ |
| budget-constrained-design |
36 |
numeric |
\ |
| budget-constrained-design |
37 |
numeric |
\ |
| budget-constrained-design |
38 |
numeric-table |
\ |
| budget-constrained-design |
53 |
textual |
- Are early stopping criteria pre-defined (not post-hoc)? |
Context checkpoint / Delta notes
Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.
1---2name: design-experiment3description: "Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness designs are modes, not strategy nodes."4metadata:5 source: https://skillmd.com/skills/yogsoth-ai/design-experiment6---78# design-experiment910## Purpose1112Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness are modes.1314## When to use / not applicable1516Use after a hypothesis has an operational construct and discriminating prediction. Not applicable when the hypothesis or measurable outcome is absent.1718## Input contract1920```yaml21mode_contracts:22 factorial: &experiment_input23 required: [falsifiable_hypothesis, outcome, factors, constraints]24 optional: [baseline, candidate_models, budget, robustness_axes]25 constraints: [analysis_plan_must_be_preregistered]26 ablation: *experiment_input27 comparison: *experiment_input28 scaling: *experiment_input29 robustness: *experiment_input30```3132## Execution protocol3334Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.35361. You MUST load skill `identify-variables` to operationalize the outcome, factors, controls, and their functional roles. You MUST load skill `enumerate-dimension-values` to define admissible levels or perturbation values.37 If the resource envelope or feasibility constraints cannot support an executable design, consider `analyze-constraints-readiness` before committing to a mode.382. Choose `factorial`, `ablation`, `comparison`, `scaling`, or `robustness` mode.393. You MUST load skill `specify-metrics` to preregister metrics, estimands, directionality, and decision thresholds. You MUST load skill `estimate-sample-size` to derive the sample or repetition requirement. You MUST load skill `select-statistical-method` to select the inference or estimation method before observing outcomes; specify power and the stopping rule.40 If the proposed metric or validator may share artifacts, labels, or assumptions with the system under test, consider `audit-validator-independence` before freezing the analysis plan.414. You MUST load skill `construct-design-matrix` to construct the runnable matrix for the selected mode. You MUST load skill `design-randomness-protocol` to define seeds, repetitions, and propagation rules. You MUST load skill `specify-execution-environment` to capture interpretation-relevant hardware, software, data, configuration, and versions. You MUST load skill `specify-reproducibility-protocol` to define and test the intended reproduction level. You MUST load skill `optimize-design-under-budget` to select a feasible information-efficient design under the declared resource envelope; include resource and failure checks.42 Once the design has produced observations and the task changes from planning to inference, `analyze-experiment-results` may be the better next tactic.4344## Mode branches4546For `comparison` or `robustness`, You MUST load skill `select-experimental-baseline` to choose a controlled baseline matched to the claim.4748- `factorial`: vary multiple factors in a structured design so main effects and interactions are estimable within the declared resource envelope.49- `ablation`: remove or replace components systematically to attribute the outcome to individual parts and suspected interactions. You MUST load skill `map-ablation-components` to define ablatable units, dependencies, and legal removal or replacement operations.50- `comparison`: evaluate a target against controlled baselines with matched confounds, compute, tuning effort, and preregistered tests.51- `scaling`: instantiate geometric or otherwise justified scale points to test how the outcome changes across the declared regime.52- `robustness`: perturb relevant conditions or inputs and measure whether the claimed effect survives the defined stress space.5354## Output contract5556```yaml57mode_contracts:58 factorial:59 produces: [factor_level_matrix, factor_level_catalog, estimands_main_effects_interactions, metric_significance_plan, sample_power_plan]60 delta_fields: [findings, decisions, uncertainties, open_questions]61 ablation:62 produces: [ablation_matrix, baseline_anchors_full_minimal, attribution_contrasts, component_interaction_plan]63 delta_fields: [findings, decisions, uncertainties, open_questions]64 comparison:65 produces: [controlled_baseline_comparison, matched_confound_controls, seed_environment_protocol, statistical_comparison_plan, reproducibility_protocol]66 delta_fields: [findings, decisions, uncertainties, open_questions]67 scaling:68 produces: [scaling_axes, geometric_scale_points, scaling_experiment_grid, curve_fit_plan, scale_budget_plan]69 delta_fields: [findings, decisions, uncertainties, open_questions]70 robustness:71 produces: [perturbation_stress_matrix, severity_axes, baseline_comparison, degradation_metrics, survival_criteria]72 delta_fields: [findings, decisions, uncertainties, open_questions]73```7475## Thresholds and quality gates7677- `experiment-design` HARD-GATE and Budget Gate remain mandatory; no exit before declared minimum yield.78- Resource and run-scale gates are relative to the declared resource envelope and eligible factor/condition space; record numerator, denominator, batch increment, stopping reason, and source references.79- Scale, ablation, comparison, and robustness coverage must reach a justified relative floor over the declared design space; do not substitute an unreasoned fixed run count.80- Factor levels, comparison baseline, statistical test, significance threshold, sample-size rationale, and stopping rule must all be explicit.81- Significance threshold must be pre-registered, not chosen post-hoc.82- Budget-constrained design must report at least one feasible design under the stated resource envelope.83- Scaling mode retains the source criterion of a geometric progression, typically 4-8 points, while evaluating coverage relative to the declared scale domain. Relative audit: declared universe = eligible scale domain; numerator = scale points instantiated; batch increment = points added per design pass; stopping reason = predeclared coverage or saturation rule; source references = scaling-design ledger and design records; direction/threshold rationale = geometric spacing preserves regime sensitivity, with 4-8 retained as the source band.8485## Failure and counterexamples8687Reject post-hoc factor selection, outcome leakage, missing control, unpowered comparison, or a design whose claimed inference exceeds its measured outcome.8889## Provenance map9091`experiment-design`, `factor-level-design`, `ablation-design`, `comparison-design`, `scaling-design`, `robustness-design`, `statistical-method-selection`, `reproducibility-protocol`, `budget-constrained-design`; mode-specific steps retained, agent dispatch removed.9293## Legacy context checkpoint / Delta notes9495Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.9697## Preserved source criteria ledger9899| source | source line | kind | source criterion |100|---|---:|---|---|101| experiment-design | 45 | textual | ## HARD-GATE |102| experiment-design | 47 | textual | Before entering this campaign, the following must be satisfied: |103| experiment-design | 64 | textual | 4. How to ensure reproducibility |104| experiment-design | 79 | textual | ## Budget Gate |105| experiment-design | 83 | numeric | \\| Micro \\| < 10 \\| 3 \\| 20 \\| Fractional factorial or single ablation \\| |106| experiment-design | 84 | numeric | \\| Small \\| 10-100 \\| 5 \\| 50 \\| Full factorial on key factors \\| |107| experiment-design | 85 | numeric | \\| Medium \\| 100-1000 \\| 8 \\| 200 \\| Multi-strategy composition \\| |108| experiment-design | 86 | numeric | \\| Large \\| > 1000 \\| Unlimited \\| Unlimited \\| Full design space exploration \\| |109| experiment-design | 95 | textual | ## Minimum Yield |110| experiment-design | 97 | textual | Every campaign invocation must produce at minimum: |111| ablation-design | 43 | textual | ## Budget Gate |112| ablation-design | 47 | numeric | \\| Systematic (leave-one-out) \\| 3-8 \\| N + 2 \\| Standard component analysis \\| |113| ablation-design | 48 | numeric | \\| Replacement \\| 3-8 \\| 2N + 2 \\| Need to distinguish "removal" vs "simplification" \\| |114| ablation-design | 49 | numeric | \\| Combinatorial (selected) \\| 4-6 \\| ~2N \\| Suspected interactions between components \\| |115| ablation-design | 50 | numeric | \\| Combinatorial (full) \\| 3-4 \\| 2^N \\| Small systems, need complete picture \\| |116| ablation-design | 51 | numeric | \\| Conditional \\| 3-6 \\| N * conditions \\| Context-dependent contributions \\| |117| comparison-design | 34 | textual | - **Fair Comparison Protocol** (Bouthillier 2021): Control all confounds, same compute budget, same tuning effort. |118| comparison-design | 47 | textual | 6. **reproducibility-protocol** (tactic) -> Ensure all results are reproducible |119| comparison-design | 50 | textual | ## Budget Gate |120| comparison-design | 54 | numeric-table | \\| Minimal \\| 1 SOTA + 1 simple \\| 1 \\| 3 \\| 6 \\| |121| comparison-design | 55 | numeric | \\| Standard \\| 2-3 baselines \\| 2-3 \\| 5 \\| 30-45 \\| |122| comparison-design | 56 | numeric | \\| Comprehensive \\| 4+ baselines \\| 3-5 \\| 5-10 \\| 100+ \\| |123| comparison-design | 57 | numeric-table | \\| Publication-ready \\| All relevant \\| 5+ \\| 10+ \\| 200+ \\| |124| scaling-design | 43 | numeric | 2. **level-specification** -> Define scale points (geometric progression, typically 4-8 points) |125| scaling-design | 49 | textual | ## Budget Gate |126| scaling-design | 53 | numeric | \\| Data scaling \\| 4-6 \\| 3 \\| 12-18 \\| Low (same model, subset data) \\| |127| scaling-design | 54 | numeric | \\| Model scaling \\| 4-8 \\| 2-3 \\| 8-24 \\| High (different model sizes) \\| |128| scaling-design | 55 | numeric | \\| Compute-optimal \\| 6-10 per iso-FLOP \\| 1-2 \\| 12-20 \\| Very high \\| |129| scaling-design | 56 | numeric | \\| Inference scaling \\| 5-10 \\| 5 \\| 25-50 \\| Low (inference only) \\| |130| robustness-design | 50 | textual | ## Budget Gate |131| robustness-design | 54 | numeric | \\| Single perturbation \\| 1 \\| 3-5 \\| 3-5 \\| Quick sanity check \\| |132| robustness-design | 55 | numeric | \\| Multi-perturbation \\| 3-5 \\| 3 each \\| 9-15 \\| Standard robustness eval \\| |133| robustness-design | 56 | numeric | \\| Adversarial sweep \\| 1 attack \\| 5-10 epsilon \\| 5-10 \\| Adversarial robustness curve \\| |134| robustness-design | 57 | numeric | \\| Comprehensive \\| 5+ types \\| 3-5 each \\| 50+ \\| Publication-ready robustness \\| |135| robustness-design | 58 | numeric-table | \\| Cross-domain \\| N domains \\| 1 \\| N \\| Transfer evaluation \\| |136| statistical-method-selection | 30 | numeric-table | \\| Normal data, 2 groups, paired \\| Paired t-test \\| |137| statistical-method-selection | 31 | numeric-table | \\| Normal data, 2 groups, unpaired \\| Welch's t-test \\| |138| statistical-method-selection | 32 | numeric-table | \\| Normal data, 3+ groups \\| ANOVA + post-hoc (Tukey HSD) \\| |139| statistical-method-selection | 33 | numeric-table | \\| Non-normal, 2 groups \\| Wilcoxon signed-rank / Mann-Whitney U \\| |140| statistical-method-selection | 34 | numeric-table | \\| Non-normal, 3+ groups \\| Kruskal-Wallis + Dunn's test \\| |141| statistical-method-selection | 36 | numeric-table | \\| Want probability of superiority \\| Bayesian comparison (Benavoli 2017) \\| |142| statistical-method-selection | 48 | textual | - Is the significance threshold pre-registered (not chosen post-hoc)? |143| reproducibility-protocol | 17 | textual | # Tactic: Reproducibility Protocol |144| reproducibility-protocol | 21 | textual | 1. **Assess Reproducibility Requirements** -> Determine level needed (exact, statistical, conceptual) |145| reproducibility-protocol | 22 | textual | 2. **seed-protocol-design** -> Design random seed strategy for all stochastic components |146| reproducibility-protocol | 24 | textual | 4. **Define Verification Plan** -> How to confirm reproducibility (re-run subset, cross-machine test) |147| reproducibility-protocol | 25 | textual | 5. **Document Non-Determinism** -> Identify and document unavoidable sources of variance |148| reproducibility-protocol | 29 | textual | \\| Reproducibility Level \\| Requirement \\| When to Use \\| |149| reproducibility-protocol | 37 | textual | - Are all random seeds documented and controllable? |150| reproducibility-protocol | 38 | textual | - Is the full software environment captured (versions, dependencies)? |151| reproducibility-protocol | 39 | textual | - Are hardware-specific non-determinisms identified (GPU atomics, cuDNN)? |152| reproducibility-protocol | 40 | textual | - Is there a verification protocol (re-run N times, check variance)? |153| reproducibility-protocol | 44 | textual | - Is there a plan for cross-machine reproducibility testing? |154| budget-constrained-design | 34 | numeric | \\| < 10 \\| One-factor-at-a-time or Plackett-Burman screening \\| |155| budget-constrained-design | 35 | numeric | \\| 10-30 \\| Fractional factorial (Resolution III-IV) \\| |156| budget-constrained-design | 36 | numeric | \\| 30-60 \\| Fractional factorial (Resolution V) or Taguchi \\| |157| budget-constrained-design | 37 | numeric | \\| 60-120 \\| Full factorial on top factors + screening on rest \\| |158| budget-constrained-design | 38 | numeric-table | \\| 120+ \\| Full factorial or RSM with replication \\| |159| budget-constrained-design | 53 | textual | - Are early stopping criteria pre-defined (not post-hoc)? |160161## Context checkpoint / Delta notes162163Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.