Training Run Triage

Training Run Triage

Purpose

Turn a large, noisy training run into an evidence-based diagnosis.

This skill is for:

• reading and interpreting training logs
• understanding why a run succeeded, plateaued, diverged, overfit, underfit, or regressed
• finding instability and bottlenecks
• extracting the main story of the run
• identifying what was missing in instrumentation
• recommending the next most informative actions

The goal is not to restate charts. The goal is to explain what happened, why it likely happened, and what to do next.

Use when

Use this skill when:

• the user shares training logs, metric screenshots, ClearML tasks, run IDs, or artifact links
• the user asks what happened in training
• a run looks degraded, unstable, surprisingly good, suspiciously flat, or too slow
• the user wants interpretation of metrics and dynamics
• the user wants postmortem analysis of an experiment
• the user wants to compare the run against expectation, previous runs, or baseline

Typical trigger phrases:

• "разбери логи обучения"
• "посмотри clearml"
• "что случилось с этим run"
• "почему деградация"
• "тут instability, разберись"
• "проинтерпретируй метрики"
• "почему обучение такое медленное"
• "почему val ведет себя странно"
• "analyze this training run"
• "triage this experiment"

Do not use when

Do not use this skill for:

• designing a training pipeline from scratch
• writing training code as the primary task
• generic performance optimization of unrelated application code
• paper analysis
• full experiment roadmap creation
• hyperparameter search planning without a concrete run to inspect

Inputs

Expected inputs:

• a concrete run, task, experiment, or a small set of runs
• the current source context for where to fetch logs and artifacts
• run metadata and available artifacts

Optional inputs:

• baseline run(s)
• expected behavior or success criteria
• screenshots of metrics
• known recent code, data, or config changes
• hypotheses already under consideration

Outputs

Always produce:

1. run verdict
2. concise run narrative or timeline
3. evidence-backed interpretation of key metrics
4. diagnosis of degradation, instability, bottlenecks, and anomalies
5. surprising findings
6. missing instrumentation or missing artifacts
7. prioritized next actions
8. optionally, a written triage note if the repository already keeps experiment analyses

Core responsibilities

This skill must not stop at "loss went down" or "val is noisy".

It must explicitly cover:

• run context reconstruction
• metric interpretation
• convergence quality
• generalization behavior
• instability detection
• efficiency and bottleneck diagnosis
• artifact inspection
• logging-quality audit
• next-step recommendation

Constraints

• If the current session already makes the run source obvious, use that source directly.
• Only ask for clarification when the target run is genuinely ambiguous.
• Verify logging integrity before trusting metric curves.
• Separate strong conclusions from plausible hypotheses.
• Prefer compact, evidence-backed findings over generic ML advice.
• End with the smallest high-signal next actions rather than a broad experiment wishlist.

Procedure

1. Identify the target run(s). Infer from the current session context which run or server/source to use. If one source is clearly implied by the conversation, use it directly. Only ask for clarification if the target run is genuinely ambiguous.

2. Collect the full run context. Fetch, inspect, and summarize:

   • run name, task ID, and timestamps
   • experiment group or tags
   • code version, commit, or branch if available
   • config and hyperparameters
   • optimizer and scheduler settings
   • batch size, accumulation, and precision mode
   • dataset, split, and version information
   • checkpoint settings
   • resume or restart information
   • host, GPU, and hardware metadata
   • console logs
   • scalar histories
   • system telemetry
   • evaluation artifacts
   • saved predictions, examples, confusion matrices, PR curves, or tables
   • related baseline runs if visible

3. Reconstruct the run timeline. Build a practical timeline:

   • startup or initialization
   • warmup
   • early learning
   • stable training region
   • plateau, overfit onset, divergence point, or slowdown point
   • checkpoint peaks
   • termination condition
   • resume events or interruptions if any

4. Validate logging integrity before interpretation. Check whether the logs themselves are trustworthy:

   • metric names are consistent
   • step or epoch alignment is correct
   • train and validation are comparable
   • smoothing is not hiding spikes
   • "best" checkpoint and displayed metric actually correspond
   • eval cadence is not too sparse or misleading
   • missing points, resets, or resumed counters are understood
   • metrics are computed on the expected subset or split
   • directionality is correct

5. Interpret optimization dynamics. Analyze:

   • train loss shape
   • validation loss shape
   • learning rate schedule and phase transitions
   • gradient norm, clipping, or overflow indicators if present
   • loss spikes, oscillation, NaNs, or sudden collapses
   • stabilization after warmup
   • signs of too-high LR, too-low LR, scheduler bugs, or bad warmup
   • effect of regularization, augmentation, label smoothing, EMA, and similar settings if inferable

6. Interpret model quality and generalization. Analyze:

   • best metric vs final metric
   • gap between train and validation
   • underfitting vs overfitting
   • slow but healthy improvement vs fake plateau
   • validation noise level
   • whether the model is genuinely learning useful structure
   • whether gains are statistically weak, noisy, or fragile
   • checkpoint selection quality
   • whether early stopping would have helped or hurt

7. Inspect class-wise or slice-wise behavior when possible. If artifacts allow, inspect:

   • per-class metrics
   • long-tail classes
   • rare-event behavior
   • threshold sensitivity
   • calibration issues
   • confusion structure
   • subgroup or slice regressions
   • headline metric looks fine but critical slice is bad

8. Inspect qualitative artifacts when available. Review:

   • sampled predictions
   • failure examples
   • attention, saliency, or visualization artifacts if present
   • confusion matrices
   • precision-recall, ROC, or calibration curves
   • bad examples or edge cases Use qualitative artifacts to refine, not replace, the metric-based conclusion.

9. Diagnose instability. Explicitly look for:

   • divergence
   • oscillation
   • late-epoch collapse
   • exploding or vanishing gradients
   • mixed-precision overflow or scaler instability
   • batch-statistics instability
   • checkpoint corruption or bad resume state
   • data ordering or sampler issues
   • extreme sensitivity to seed or eval subset noise
   • unstable thresholded metrics

10. Diagnose efficiency and bottlenecks. Analyze available telemetry and logs for:

• throughput over time
• GPU utilization
• memory pressure or OOM proximity
• data loader starvation
• CPU bottlenecks
• disk, network, or object-store I/O stalls
• evaluation overhead
• checkpoint save overhead
• logging overhead
• synchronization or distributed-training stalls
• unusually slow startup or compile time
• expensive augmentations or preprocessing Separate bad model learning from slow system execution.

11. Compare against baseline or neighboring runs when possible. If related runs are available:

• compare config deltas
• compare data version deltas
• compare code changes
• compare metric trajectories, not just final values
• isolate which change likely moved the outcome
• identify whether the current run is genuinely worse or only appears worse due to noise, cadence, or metric-definition drift

12. Identify surprising findings. Surface facts that are easy to miss, for example:

• best checkpoint happened much earlier than expected
• train improved while validation never really moved
• validation is too noisy to support strong conclusions
• apparent degradation came from changed metric definition
• run spent large time outside actual training
• one class or slice drove the whole metric movement
• the run was likely bottlenecked by evaluation, not optimization
• resume likely altered behavior
• last checkpoint is materially worse than best
• improvement exists but is operationally irrelevant

13. Audit missing instrumentation. Explicitly list what was missing and would have made diagnosis materially better, such as:

• gradient norm or clipping stats
• LR logging per step or phase
• throughput and samples per second
• data loader time vs step time
• per-class metrics
• confusion matrices
• calibration or threshold curves
• train/validation split identifiers and dataset version
• checkpoint summary table
• qualitative sample panels
• system telemetry
• augmentation statistics
• optimizer state summaries
• memory usage over time
• eval latency breakdown
• seed and reproducibility metadata
• exact code snapshot or commit

14. Produce a ranked diagnosis. Separate:

• strong conclusions supported by evidence
• plausible hypotheses
• unknowns blocked by missing logs

15. Produce a next-step plan. Recommend only the most informative next actions, prioritized by signal-to-cost ratio:

• small rerun with logging fixes
• one targeted ablation
• one config correction
• one bottleneck removal
• one validation improvement Avoid generic "try many hyperparameters" advice.

Decision rules

Evidence rule

Every major conclusion must be tied to observed metrics, logs, telemetry, or artifacts. Do not speculate without labeling the statement as a hypothesis.

Run-story rule

Do not report isolated facts only. Construct a coherent narrative of the run.

Metric-integrity rule

Before trusting a metric, verify that it is computed, aligned, and displayed correctly.

Bottleneck rule

Separate optimization pathology from systems pathology. A slow run is not necessarily a bad-learning run, and a bad-learning run is not necessarily system-bound.

Comparison rule

When baselines exist, interpret deltas relative to the changed config, data, or code, not only by absolute final metric.

Missing-data rule

Always state what cannot be concluded due to missing instrumentation or artifacts.

Actionability rule

End with a short ranked list of next actions that maximally reduce uncertainty.

Optional subagent use

For large runs or rich experiment trackers, explicitly use bounded subagents:

• one run-fetcher subagent: gathers metrics, console logs, artifacts, config, system telemetry, and relevant neighboring runs

• one metric-forensics subagent: interprets curves, instability patterns, generalization behavior, and checkpoint dynamics

• one systems-bottleneck subagent: inspects throughput, utilization, I/O stalls, eval overhead, and training-system efficiency

Wait for all bounded analyses, then merge them into one final triage report.

Do not use multiple subagents for a tiny local log snippet.

References

Use the supporting references when needed:

• references/metric-interpretation-checklist.md
• references/instability-signals.md
• references/missing-logging-checklist.md
• references/systems-bottleneck-checklist.md
• references/experiment-comparison-template.md

Suggested report structure

The final triage report should contain:

• Run summary
• Verdict
• Timeline
• Key metrics and interpretation
• Generalization or convergence diagnosis
• Instability diagnosis
• Bottleneck diagnosis
• Surprising findings
• Missing instrumentation
• Ranked next actions
• Confidence or unknowns

Optional persisted document

If the repository already has a notes or experiment-analysis convention, write a note such as:

• docs/analyses/training-runs/<run-slug>.md
• notes/experiments/<run-slug>.md

Otherwise return the report directly in chat.

Definition of done

• The run context was reconstructed from logs, artifacts, config, and metadata.
• Metric interpretation is tied to concrete evidence.
• Learning pathology and systems pathology are separated clearly.
• Missing instrumentation is explicit and actionable.
• The next-step list is short, prioritized, and designed to reduce uncertainty.

Final response format

Return a compact answer in this structure:

• Run: ...
• Verdict: ...
• Timeline: ...
• Key metric interpretation: ...
• Instability or degradation: ...
• Bottlenecks: ...
• Surprising findings: ...
• Missing logs or artifacts: ...
• Next actions: ...
• Confidence or unknowns: ...

Positive examples

Use this skill for:

• "разбери run в clearml"
• "что случилось с этим обучением"
• "почему деградация после 8 эпохи"
• "посмотри метрики и скажи, норм ли run"
• "где здесь instability"
• "почему run такой медленный"
• "analyze this training run"

Negative examples

Do not use this skill for:

• "сделай план новой архитектуры"
• "напиши training loop"
• "ускорь этот не-ML модуль"
• "прочитай статью и предложи идею"
• "пофикси баг в инференсе"