STANDARD OPERATING PROCEDURE
Purpose
Rapidly triage ML training incidents (instability, divergence, degraded metrics) and deliver validated remediations with traceable evidence.
Triggers
- Positive: Failing/unstable training runs, unexplained metric drops, NaNs/exploding gradients, data/label issues, reproducibility gaps.
- Negative: New model development without incident (route to
ml-expert) or lightweight prototyping (route to ml).
Guardrails
- Structure-first: ensure
SKILL.md, README, examples/, tests/, resources/, and agents/ exist; create missing docs before work.
- Constraint extraction: clarify environment (hardware/framework), data provenance, metric targets, and incident timeline.
- Validation discipline: reproduce issue, isolate variables (data/model/optim), run minimal change tests; adversarially probe for leakage and nondeterminism.
- Confidence ceiling enforced (inference/report 0.70; research 0.85; observation/definition 0.95) with evidence per finding.
- Safety: preserve checkpoints/logs; avoid destructive changes; keep rollback path ready.
Execution Phases
- Intake & Evidence Gathering
- Collect logs, metrics, configs, seeds, hardware info, and recent code changes.
- Confirm baseline vs expected behavior and incident start time.
- Reproduction & Isolation
- Reproduce on smallest dataset slice; fix seeds; disable randomness.
- Binary-search variables: data batches, preprocessing, model changes, optimizer settings.
- Hypothesis & Experiment Plan
- Form hypotheses (data corruption, label leakage, optimizer instability, precision issues).
- Plan targeted experiments with success/fail criteria.
- Fix & Validation
- Implement minimal fixes; run controlled tests (train/val curves, gradient norms, loss stats).
- Validate against performance/latency targets; ensure no regression on baseline metrics.
- Handoff & Prevention
- Document root cause, applied fixes, and remaining risks.
- Add monitors/tests to prevent recurrence; package rollback instructions.
Output Format
- Incident summary and constraints.
- Reproduction steps, hypotheses, and experiments run.
- Fixes applied with before/after metrics.
- Prevention plan and owners.
- Confidence statement with ceiling.
Validation Checklist
VCL COMPLIANCE APPENDIX (Internal)
[[HON:teineigo]] [[MOR:root:H-T-A]] [[COM:Hata+Teshis+Analiz]] [[CLS:ge_skill]] [[EVD:-DI]] [[ASP:nesov.]] [[SPC:path:/skills/specialists/ml-training-debugger]]
[[HON:teineigo]] [[MOR:root:E-P-S]] [[COM:Epistemik+Tavan]] [[CLS:ge_rule]] [[EVD:-DI]] [[ASP:nesov.]] [[SPC:coord:EVD-CONF]]
Confidence: 0.74 (ceiling: inference 0.70) - SOP rebuilt with prompt-architect constraint discipline and skill-forge structure/validation rules.
1---2name: ml-training-debugger3description: Diagnose and stabilize ML training runs, recover from failures, and deliver validated fixes.4---5
6
7## STANDARD OPERATING PROCEDURE
8
9### Purpose
10Rapidly triage ML training incidents (instability, divergence, degraded metrics) and deliver validated remediations with traceable evidence.
11
12### Triggers
13- **Positive:** Failing/unstable training runs, unexplained metric drops, NaNs/exploding gradients, data/label issues, reproducibility gaps.
14- **Negative:** New model development without incident (route to `ml-expert`) or lightweight prototyping (route to `ml`).
15
16### Guardrails
17- Structure-first: ensure `SKILL.md`, `README`, `examples/`, `tests/`, `resources/`, and `agents/` exist; create missing docs before work.
18- Constraint extraction: clarify environment (hardware/framework), data provenance, metric targets, and incident timeline.
19- Validation discipline: reproduce issue, isolate variables (data/model/optim), run minimal change tests; adversarially probe for leakage and nondeterminism.
20- Confidence ceiling enforced (inference/report 0.70; research 0.85; observation/definition 0.95) with evidence per finding.
21- Safety: preserve checkpoints/logs; avoid destructive changes; keep rollback path ready.
22
23### Execution Phases
241. **Intake & Evidence Gathering**
25 - Collect logs, metrics, configs, seeds, hardware info, and recent code changes.
26 - Confirm baseline vs expected behavior and incident start time.
272. **Reproduction & Isolation**
28 - Reproduce on smallest dataset slice; fix seeds; disable randomness.
29 - Binary-search variables: data batches, preprocessing, model changes, optimizer settings.
303. **Hypothesis & Experiment Plan**
31 - Form hypotheses (data corruption, label leakage, optimizer instability, precision issues).
32 - Plan targeted experiments with success/fail criteria.
334. **Fix & Validation**
34 - Implement minimal fixes; run controlled tests (train/val curves, gradient norms, loss stats).
35 - Validate against performance/latency targets; ensure no regression on baseline metrics.
365. **Handoff & Prevention**
37 - Document root cause, applied fixes, and remaining risks.
38 - Add monitors/tests to prevent recurrence; package rollback instructions.
39
40### Output Format
41- Incident summary and constraints.
42- Reproduction steps, hypotheses, and experiments run.
43- Fixes applied with before/after metrics.
44- Prevention plan and owners.
45- Confidence statement with ceiling.
46
47### Validation Checklist
48- [ ] Issue reproduced with fixed seeds and minimal data slice.
49- [ ] Hypotheses tested; experiments documented.
50- [ ] Metrics reported per split; gradients/loss inspected where relevant.
51- [ ] Regression checks executed; rollback path documented.
52- [ ] Confidence ceiling stated.
53
54## VCL COMPLIANCE APPENDIX (Internal)
55[[HON:teineigo]] [[MOR:root:H-T-A]] [[COM:Hata+Teshis+Analiz]] [[CLS:ge_skill]] [[EVD:-DI<gozlem>]] [[ASP:nesov.]] [[SPC:path:/skills/specialists/ml-training-debugger]]
56
57[[HON:teineigo]] [[MOR:root:E-P-S]] [[COM:Epistemik+Tavan]] [[CLS:ge_rule]] [[EVD:-DI<gozlem>]] [[ASP:nesov.]] [[SPC:coord:EVD-CONF]]
58
59
60Confidence: 0.74 (ceiling: inference 0.70) - SOP rebuilt with prompt-architect constraint discipline and skill-forge structure/validation rules.
majiayu000