Plan Experiment

Plan Experiment

Design a structured experiment plan before burning compute. Produce a prioritized ordering of model candidates with hypothesis, HP search space, compute estimates, and success criteria.

Invocation

Arguments ($ARGUMENTS) are interpreted as:

• --task — modeling task type (classification, regression, ranking, forecasting)
• --data PATH|DATASET — dataset path or identifier
• --metric METRIC — primary evaluation metric (e.g., f1, rmse, ndcg@10)
• --budget GPU_HOURS — total compute budget in GPU-hours (default: ask)
• --max-candidates N — maximum model families to evaluate (default: 5)
• --horizon DURATION — prediction window if applicable

Target: $ARGUMENTS

Your responsibilities

1. Translate objective into a falsifiable ML hypothesis

State the hypothesis in the form:

"We believe [model family] will achieve [metric] ≥ [threshold] on [dataset] within [compute budget], because [rationale]."

If the objective or metric is missing, invoke review-target before proceeding.

2. Select 2–5 model candidates

Always include:

• Candidate #1: simplest possible baseline (logistic regression, linear regression, decision tree, or frequency prior)
• Additional candidates ranked by expected signal per compute dollar

Use references/hp-search-strategies.md for model family guidance. Cap at --max-candidates (default 5).

3. Define HP search space per candidate

For each candidate, specify:

• hyperparameter names, ranges, and scales (log, linear, categorical)
• search strategy (grid, random, Bayesian, successive halving)
• number of trials and early-stopping rule

Use references/hp-search-strategies.md for family-specific default ranges.

4. Estimate compute requirements

For each candidate:

• estimate GPU-hours using references/compute-estimation-guide.md
• convert to approximate cost (use $3/GPU-hour as default cloud estimate unless user specifies)
• flag if total budget is exceeded; suggest pruning strategy

5. Set success/failure criteria and time-box

Define:

• primary success criterion (metric threshold to beat baseline)
• failure trigger (stop condition per candidate: no improvement after N trials, loss diverges, etc.)
• wall-clock time-box per candidate

6. Recommend experiment ordering

Order candidates by:

1. cheapest/fastest first (quick signal)
2. successive halving: allocate more budget to survivors
3. skip candidates if earlier results already beat success criterion

Output format

Experiment Plan
===============
Objective: <business outcome>
Hypothesis: <falsifiable statement>
Task: <task type>
Primary metric: <metric>
Dataset: <path/ID>
Total budget: <GPU-hours> (~$<cost>)

Model Candidates (ordered):
| # | Model family      | Rationale                | Est. GPU-hrs | Est. cost |
|---|-------------------|--------------------------|--------------|-----------|
| 1 | <baseline>        | <why>                    | <h>          | $<cost>   |
| 2 | <candidate>       | <why>                    | <h>          | $<cost>   |

HP Search Space:
Candidate 1 — <model>:
  <param>: range=[<lo>, <hi>], scale=<log|linear>, search=<strategy>

Success criteria:
- Primary: <metric> ≥ <threshold> vs. baseline
- Failure trigger per candidate: <condition>
- Time-box per candidate: <duration>

Experiment ordering:
1. Run <candidate> — expected <duration>, budget <GPU-hrs>
2. If criterion met: STOP and proceed to feature-engineer / check-data-pipeline
   If not: run <next candidate>

Decision: GO | NO-GO (missing: <what>)

JSON artifact

Write plan-experiment.json to --out-dir (or ./ if invoked standalone) following the schema in ../../references/schemas.md. Use vocabulary from ../../references/vocabulary.md.

Key fields to populate from the plan output:

• decision: GO when plan is complete and ready; NO-GO when blocking input is missing
• hypothesis, candidates, success_criteria, experiment_order
• findings: one entry per blocker or gap (severity blocker for missing metric/dataset, medium for warnings)
• next_commands: ordered experiment run commands

Quick heuristics

• No baseline → add one before any deep learning candidate; reject plan otherwise
• Budget < 1 GPU-hour for all candidates → suggest CPU-only models or reduce search trials
• Dataset < 10 K rows → warn against deep learning; prefer tree ensembles or linear models
• No metric chosen → request metric or invoke review-target
• User says "try everything" → cap at 5 model families / 50 trials total; apply successive halving
• All candidates are neural networks → add gradient-boosted trees as Candidate #2 (strong baseline for tabular data)

Stop conditions

Stop when:

• experiment plan with all sections is complete and ordering is defined, or
• blocking input (metric, dataset, task type) is missing and explicitly requested.

Additional resources

• references/hp-search-strategies.md — HP ranges by model family and search algorithm selection
• references/compute-estimation-guide.md — GPU throughput heuristics and cost formulas