jurgendn Model Eval Error Analysis
Do not stop at one headline metric.
Intake
Collect:
- task, dataset, split, metric, and baseline;
- model outputs or failure examples if available;
- aggregate numbers with seeds/uncertainty;
- deployment or paper claim the evaluation must support;
- known risky slices.
Workflow
- Verify the metric implementation and data split.
- Report the headline result with provenance.
- Slice performance by relevant factors such as:
- difficulty
- length / scale
- class imbalance
- domain shift
- prompt family
- robustness setting
- Inspect failures manually and cluster them.
- Separate:
- random noise
- systematic failure modes
- evaluator bugs
- Compare against the strongest baseline on the same slices.
- End with concrete next actions.
Rules
- Never smooth away inconvenient variation.
- If confidence intervals or multiple seeds matter, say so.
- A higher average may still hide a worse model for important slices.
Output shape
Return:
- Metrics checked
- Slice analysis
- Failure taxonomy
- Suspected causes
- Recommended fixes or follow-up experiments