Orchestrate E2e

Orchestrate E2E

Run a complete ML lifecycle with explicit gates, evidence capture, and promotion decisions.

Invocation

Arguments ($ARGUMENTS) are interpreted as:

• <objective> — plain-language business objective and model outcome
• --data SOURCE — dataset path, HF dataset, or DB target
• --label-col COLUMN — training target column
• --metric METRIC — primary selection metric (for example f1, rmse, auprc)
• --train-cmd COMMAND — canonical training command to execute
• --eval-target — checkpoint/model to evaluate
• --baseline — baseline checkpoint or results file for delta checks
• --out-dir DIR — output directory for run artifacts (default: ./reports/e2e-run)
• --budget-hours N — wall-clock budget
• --max-retries N — max retries after failed runs (default: 2)

Target: $ARGUMENTS

Your responsibilities

1. Establish the run contract

Before execution, generate a run_id (UUID or timestamp slug, e.g., run-20260227-143022) and create the --out-dir. Write a compact contract:

• Objective and business outcome
• Data source and label
• Primary metric and minimum acceptable threshold
• Runtime and retry budget
• Promotion criteria

If any contract field is missing, ask for the minimum clarification and continue.

2. Build a gate plan

Use references/lifecycle-gates.md and define pass/fail criteria for each stage:

1. target readiness
2. experiment plan
3. non-ML baseline
4. dataset quality
5. data pipeline integrity
6. training stability
7. evaluation quality
8. interpretability and bias
9. promotion decision

Never skip gates without explicitly recording why.

NO-GO halt policy: if any gate returns NO-GO, immediately halt — do not execute subsequent gates. Write all artifacts collected so far, generate the partial HTML report, and return a final NO-GO decision that names the blocking gate and its actionable fixes.

3. Execute pre-training gates

Pass --out-dir <out-dir> --run-id <run_id> to every sub-skill so each writes its JSON artifact to the shared run directory.

Run and record:

• review-target — gate 1
• plan-experiment — gate 2
• build-baseline — gate 3
• check-dataset-quality — gate 4
• check-data-pipeline — gate 5

Apply the NO-GO halt policy after each gate.

4. Execute training gate (gate 6)

When --train-cmd is provided:

• Invoke train-model with the training command and any HP config from plan-experiment.json (if available in --out-dir)
• Pass --out-dir <out-dir> --run-id <run_id> so all sub-artifacts land in the shared directory
• train-model internally handles babysit-training monitoring and check-failed-run on failure
• Apply the NO-GO halt policy after train-model completes

When --train-cmd is missing, request it and wait.

5. Execute evaluation gate (gate 7)

Run check-eval on the selected checkpoint/model:

• Compare against baseline when provided; if build-baseline.json exists in --out-dir, pass it as --baseline automatically so the non-ML baseline score is the comparison floor
• Record regressions and improvements
• Produce a clear pass/fail against metric thresholds
• Apply the NO-GO halt policy after check-eval completes

5b. Execute interpretability gate (gate 8)

Run explain-model on the promoted checkpoint:

• Apply the NO-GO halt policy after explain-model completes

5c. Demonstrate value (optional)

If --business-context is provided or a business context is clearly inferable from the objective, invoke demonstrate-value:

• Pass --out-dir <out-dir> --run-id <run_id> and --business-context as appropriate
• This step does not affect the promotion decision — it is informational only
• If demonstrate-value fails, log the error and continue to the promotion step

5d. Recommend new approaches (always)

After all evaluation and interpretability gates complete, invoke recommend-new-approaches:

• Pass --out-dir <out-dir> --run-id <run_id> so it picks up all artifacts automatically
• This step does not affect the promotion decision — it is always informational and forward-looking
• If recommend-new-approaches fails, log the error and continue to the promotion step
• The recommendations should appear in the final run summary as "Next Experiments"

6. Issue promotion decision

Produce a single decision:

• GO: all required gates pass and quality bar is met
• NO-GO: any blocker remains or quality threshold is missed

Include confidence and top risks.

7. Write run artifacts

Write artifacts listed in references/artifact-contract.md, including:

• gate status table
• evidence links/paths
• recommended next commands

After all gates complete:

1. Write run-summary.json to --out-dir (schema in ../../references/schemas.md)
2. Generate the HTML report: uv run plugins/agentic-ml/report-viewer/generate_report.py <out-dir>
3. Report the report.html path in final output

8. Keep control until terminal state

Do not hand back partial progress. Continue until:

• a final GO/NO-GO decision is delivered, or
• a hard blocker requires user input that cannot be inferred.

Output format

E2E Run Summary
===============
Objective: <...>
Data: <...>
Label: <...>
Primary metric: <...>

Gate Status:
1) Target readiness:       PASS|FAIL|SKIPPED
2) Experiment plan:        PASS|FAIL|SKIPPED
3) Non-ML baseline:        PASS|FAIL|SKIPPED
4) Dataset quality:        PASS|FAIL|SKIPPED
5) Data pipeline:          PASS|FAIL|SKIPPED
6) Training stability:     PASS|FAIL|SKIPPED
7) Evaluation quality:     PASS|FAIL|SKIPPED
8) Interpretability/bias:  PASS|FAIL|SKIPPED
9) Promotion decision:     GO|NO-GO

Decision: GO|NO-GO
Confidence: high|medium|low

Top blockers/risks:
1) ...
2) ...

Report: <out-dir>/report.html

Next commands:
- ...

Quick heuristics

• Dataset gate fails with leakage → do not proceed to pipeline or training; fix leakage first or the entire run is invalid
• Pipeline gate fails on label shift → most likely cause is seq2seq target not shifted by 1; check before retrying
• Training diverges in first 100 steps → do not burn budget retrying; pause and diagnose with check-failed-run
• Eval metric matches training metric suspiciously closely → re-check dataset gate for cross-split leakage
• Promotion threshold not set → default to "statistically significant improvement over baseline"; warn user and document assumption
• Baseline outperforms ML model → re-examine whether ML is needed (Rule 1 of Google's Rules of ML); surface as high finding
• Budget exceeded before eval → prefer a partial evaluation over none; document truncation in the run artifact

Stop conditions

Stop when any of these is true:

• a gate returns NO-GO — halt immediately, write partial artifacts, generate partial HTML report
• all 9 gates complete and the final promotion decision is delivered
• user input is required for an explicit policy decision (for example, changing threshold or budget)

Additional resources

• references/lifecycle-gates.md — stage gates and pass/fail evidence
• references/artifact-contract.md — required run artifact structure