Post Training Eval Report

Post-Training Eval Report

Purpose

Turn noisy post-training outputs into a correct engineering conclusion.

This skill is for:

• validation result analysis
• visualization review
• comparison of checkpoints, runs, or ablations
• interpretation of raw prediction artifacts
• identifying what actually improved
• identifying what is confounded, weak, or inconclusive
• deciding what to repeat, what to discard, and what to instrument next

The goal is not to restate charts. The goal is to answer:

• what is real signal
• what is likely noise or artifact
• what is operationally relevant
• what should happen next

Use when

Use this skill when:

• the user shares validation metrics, plots, screenshots, tables, prediction artifacts, or experiment folders
• the user wants a post-training result analysis
• several runs, checkpoints, or ablations must be compared
• a metric improved and the user wants to know whether the gain is real
• visualizations exist and need to be interpreted
• results look inconsistent, surprising, or too good or too bad to trust at face value

Typical trigger phrases:

• "разбери валидацию"
• "посмотри визуализации"
• "что реально улучшилось"
• "сравни эти эксперименты"
• "собери инженерный вывод по результатам"
• "что здесь confounded"
• "какие выводы из этих абляций"
• "analyze these validation results"
• "prepare an ablation report"
• "interpret these plots and metrics"

Do not use when

Do not use this skill for:

• training-run log analysis during the run
• debugging the training loop itself
• generic bug fixing
• paper reading
• architecture planning
• model implementation
• generic product analytics unrelated to model evaluation

If the main question is "what happened during training", use training-run-triage instead.

Inputs

Expected inputs:

• one or more post-training result sets
• validation metrics, plots, tables, or artifacts
• experiment context sufficient to compare runs

Optional inputs:

• baselines
• checkpoints
• prediction dumps
• confusion matrices
• PR, ROC, or calibration curves
• qualitative sample panels
• threshold sweeps
• evaluation config or postprocessing config
• code version, data version, or split version
• expected success criteria

Outputs

Always produce:

1. evaluation verdict
2. comparison summary across runs, checkpoints, or ablations
3. interpretation of what changed and why it likely changed
4. diagnosis of confounders and weak conclusions
5. slice-level, class-level, or qualitative insights when available
6. missing analyses, missing visualizations, or missing instrumentation
7. ranked next repeats, reruns, ablations, or reporting fixes

Core responsibilities

This skill must not stop at "metric improved by X".

It must explicitly answer:

• is the improvement real, fragile, or inconclusive
• what likely drove the movement
• what changed besides the intended variable
• what slices or classes benefited or regressed
• whether the plots and metrics are trustworthy
• whether the evaluation setup itself is comparable
• what needs to be repeated or rerun to confirm the result
• what additional visuals or summaries are needed

Constraints

• Check evaluation comparability before interpreting deltas.
• Distinguish intended variables from hidden changed variables.
• Treat qualitative outputs as evidence, not decoration.
• Judge operational relevance, not just metric direction.
• End with the cheapest high-signal next actions instead of a broad experiment wishlist.

Procedure

1. Identify the comparison target. Determine whether the task is about:

   • one checkpoint
   • one run vs baseline
   • multiple ablations
   • multiple seeds
   • multiple postprocessing variants
   • threshold or calibration analysis
   • qualitative result review

2. Gather full evaluation context. Collect:

   • run IDs or checkpoint IDs
   • config and postprocessing settings
   • dataset, split, and version info
   • metric definitions
   • thresholding rules
   • checkpoint selection rule
   • eval cadence or eval script version if available
   • baseline and neighboring experiment context
   • qualitative outputs and plots
   • prediction artifacts if accessible

3. Validate evaluation integrity before trusting the conclusions. Check:

   • same dataset and split were used
   • same metric definition and implementation were used
   • same thresholding, decoding, or postprocessing rules were used
   • same checkpoint-selection rule was used
   • no silent change in evaluation subset
   • no leakage, duplicate contamination, or stale-output issue is visible
   • visualizations and tables correspond to the same underlying run or checkpoint
   • reported "best" values actually match the saved outputs

4. Normalize the comparison. Before interpreting deltas, separate:

   • intended experimental variable
   • unintended changed variables
   • hidden differences such as seed, data version, filtering, threshold, eval frequency, decoding, TTA, ensembling, batch size, or selected checkpoint

5. Interpret headline metrics carefully. For each main metric:

   • quantify movement relative to baseline
   • judge whether the change is large, marginal, or noise-like
   • assess operational relevance, not just statistical direction
   • distinguish best-vs-last checkpoint behavior
   • distinguish average gain from gain concentrated in one slice

6. Analyze robustness of the conclusion. Inspect whether the result is:

   • stable across seeds
   • stable across checkpoints near the best point
   • stable across thresholds
   • stable across important slices, classes, or scenarios
   • dependent on a single favorable setup
   • likely to vanish under a repeated run

7. Analyze class-wise and slice-wise behavior. When artifacts allow, inspect:

   • per-class metrics
   • rare or long-tail classes
   • important business slices
   • difficult conditions
   • borderline cases
   • false-positive-heavy vs false-negative-heavy regions
   • whether a headline gain hides a critical regression

8. Analyze qualitative artifacts. Review:

   • representative good predictions
   • representative bad predictions
   • changed failure modes
   • "looked better visually but metric did not move" cases
   • "metric improved but outputs look suspicious" cases
   • error taxonomies that the metrics do not fully reveal

9. Analyze visualizations as evidence, not decoration. For every plot or chart, ask:

   • what exact question does it answer
   • is the scale or normalization misleading
   • does it hide support size or variance
   • does it overstate tiny improvements
   • is there a more informative alternative visualization

10. Identify confounders explicitly. Look for:

• more than one variable changed
• changed seed with no repeated runs
• changed split or filtering
• changed checkpoint selection
• changed decoding, thresholding, or postprocessing
• changed evaluation metric implementation
• changed data cleaning or leakage profile
• changed sample count or coverage
• non-comparable visual panels

11. Assess whether the gain is causal, suggestive, or unproven. Separate conclusions into:

• strong evidence
• plausible but confounded
• inconclusive or needs rerun
• likely artifact or misleading gain

12. Generate missing post-training analyses when needed. If the required raw data exists, create or request:

• confusion matrices
• per-class tables
• slice breakdowns
• threshold sweeps
• calibration plots
• PR or ROC curves
• best-vs-last checkpoint comparison
• support-size-aware tables
• error galleries or qualitative panels
• ablation summary tables
• cost-vs-quality tradeoff plots

13. Audit what is missing. Explicitly list missing elements that block a strong conclusion, such as:

• no baseline
• no repeated seeds
• no prediction artifacts
• no slice metrics
• no threshold sweep
• no confusion matrix
• no checkpoint comparison
• no support counts
• no qualitative examples
• no exact eval config snapshot
• no postprocessing record
• no evaluation script or version traceability

14. Produce a ranked engineering conclusion. The report must say:

• what likely improved
• what likely did not improve
• what got worse
• what is confounded
• what is surprising
• what to repeat
• what to stop doing

15. End with the cheapest high-signal next actions. Prefer:

• one repeat with fixed confounders
• one extra visualization or slice analysis
• one threshold or calibration sweep
• one seed repeat
• one baseline re-run under identical eval settings
• one qualitative error review on the deciding slice Avoid vague advice like "try more experiments".

Decision rules

Integrity-first rule

Never interpret result quality before checking whether the evaluation setup is actually comparable.

Causality rule

Do not attribute improvement to the intended change if other meaningful variables also changed.

Support rule

Any conclusion drawn from a metric should consider support size, slice coverage, and variance or noise.

Operational relevance rule

A metric gain is not automatically a meaningful gain. Judge whether it matters for the intended use case.

Visualization rule

Treat plots as evidence only if they clarify a concrete question. Do not let attractive visuals replace actual comparison logic.

Confounder rule

If multiple variables changed and no repeat exists, mark the conclusion as confounded unless the evidence is unusually strong.

Actionability rule

End with a short ranked list of reruns, comparisons, or analyses that best reduce uncertainty.

Optional subagent use

For large result bundles, explicitly use bounded subagents:

• one result-fetcher subagent: gathers metrics, tables, configs, prediction artifacts, plots, baselines, and relevant checkpoints

• one metric-synthesis subagent: interprets metric deltas, slice behavior, threshold dependence, and checkpoint comparisons

• one qualitative-review subagent: inspects visual examples, changed failure modes, and whether the qualitative evidence agrees with the metrics

• one confounder-audit subagent: checks whether runs are truly comparable and enumerates hidden variable changes

Wait for the bounded analyses, then merge them into one final report.

Do not use multiple subagents for a tiny local table.

References

Use the supporting references when needed:

• references/comparison-integrity-checklist.md
• references/confounder-checklist.md
• references/visualization-review-checklist.md
• references/slice-analysis-template.md
• references/evaluation-report-template.md

Suggested report structure

The final report should contain:

• Scope of comparison
• Evaluation verdict
• Main metric movement
• What likely improved
• What likely regressed
• Confounders or comparability issues
• Slice, class, or threshold findings
• Qualitative findings
• Surprising facts
• Missing analyses or visuals
• Ranked next actions
• Confidence or unknowns

Optional persisted document

If the repository or workspace already keeps analysis notes, write a note such as:

• docs/analyses/evals/<slug>.md
• notes/ablations/<slug>.md
• reports/post-training/<slug>.md

Otherwise return the report directly in chat.

Definition of done

• Evaluation comparability was checked before interpreting gains.
• The report separates real signal from confounded or inconclusive signal.
• Slice, qualitative, and support-aware evidence are used when available.
• Missing analyses and missing artifacts are explicit.
• Next actions are short, ranked, and designed to reduce uncertainty.

Final response format

Return a compact answer in this structure:

• Scope: ...
• Verdict: ...
• Main metric interpretation: ...
• What likely improved: ...
• What likely regressed: ...
• Confounders: ...
• Slice / qualitative findings: ...
• Surprising facts: ...
• Missing analyses / visuals: ...
• Next actions: ...
• Confidence / unknowns: ...

Positive examples

Use this skill for:

• "разбери нашу валидацию"
• "посмотри визуализации и скажи, что реально изменилось"
• "сравни эти абляции"
• "собери report по результатам после тренировки"
• "что здесь confounded"
• "prepare an evaluation report from these raw results"
• "which ablation actually helped"

Negative examples

Do not use this skill for:

• "что случилось во время обучения"
• "напиши training loop"
• "ускорь dataloader"
• "сделай план архитектурного изменения"
• "прочитай статью и предложи идею"