Model-Evaluation Skill
Purpose
This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right
metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It
emits a per-case metric table that the publication statistics build on, and gates the metric choice
against Metrics Reloaded (Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024.
It sits between /model-validation (which audits the split / design) and /analyze-stats (which owns
the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC,
ECE of a softmax head); /analyze-stats owns DeLong / NRI / IDI / decision curves / MRMC. Like
/analyze-stats, it generates and executes code on your predictions — numbers are never hand-typed.
When to use
- You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and
subgroup slices, plus a per-case table for the manuscript statistics.
When NOT to use
- Auditing the validation design / leakage →
/model-validation.
- DeLong / NRI / IDI / decision curves / MRMC reader study →
/analyze-stats.
- Building / training the model →
/model-scaffold; LLM / MLLM → /mllm-eval.
- Figure rendering →
/make-figures.
Workflow
Phase 1 — Fix the analysis unit and the task
State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must
respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as
per-patient.
Phase 2 — Compute task-correct metrics
Generate evaluation code that computes, on the held-out predictions:
- segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global
mean, with bootstrap 95% CIs.
- classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV at
the deployment prevalence (not a balanced set).
- detection: FROC / mAP with the IoU match criterion stated.
- interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation
metrics above plus the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a
target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time
(see the metric guide; the study design is in
/design-study + /model-validation).
- generative / synthesis (image generation or modification): full-reference similarity
(MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), plus a
downstream-task evaluation — image quality is not clinical utility (Park et al., Radiol Med 2024).
For multiclass classification, state the aggregation scheme (one-vs-rest / macro / micro /
pairwise / Obuchowski); time-to-event discrimination (Harrell's C, time-dependent ROC) is handed
to
/analyze-stats.
Add calibration (reliability diagram / ECE) and subgroup slices (the Model Card Factors).
See ${CLAUDE_SKILL_DIR}/references/metric_guide.md. Emit a per-case CSV for /analyze-stats.
Phase 3 — Gate the metric choice (deterministic)
python3 ${CLAUDE_SKILL_DIR}/scripts/check_metric_reporting.py \
--report results.md --task segmentation|classification|detection|interactive|generative --strict
PIXEL_ACCURACY_SEG / NO_BOUNDARY_METRIC / ACCURACY_ONLY / DETECTION_METRIC_MISSING must be zero.
Phase 4 — Hand off
The per-case table → /analyze-stats (DeLong / NRI / IDI / decision curves, publication tables);
figures → /make-figures; the numbers + subgroup performance → /model-card; Methods/Results →
/write-paper; compliance → /check-reporting.
Anti-Hallucination
- Never fabricate a metric value. Every number comes from executed code on the supplied predictions;
if predictions or ground truth are missing, say so and stop — do not invent a result.
- Never report pixel/voxel accuracy for segmentation or bare accuracy under imbalance — the gate
flags these; report Dice + a boundary metric, or AUROC + AUPRC with CIs.
- Never report a per-lesion metric as if it were per-patient — respect the analysis unit.
- If a metric definition or its CI method is uncertain, flag
[VERIFY] and ask.
Deterministic gate
scripts/check_metric_reporting.py — flags a task-metric mismatch / missing uncertainty (stdlib,
network-free). Reproducible challenge: bash ${CLAUDE_SKILL_DIR}/scripts/metric_reporting_challenge/verify.sh.
Reference Files
Load on demand (keep SKILL.md short):
${CLAUDE_SKILL_DIR}/references/metric_guide.md — operational checklist: the task-correct metric
per task (segmentation Dice + HD95/NSD per structure; classification AUROC + AUPRC + sens/spec at
deployment prevalence; detection FROC/mAP with a stated IoU), plus calibration, subgroup slices,
run-variance, and the per-case CSV hand-off.
${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md — the standards grounding behind
those choices: the Metrics Reloaded task-fingerprint principle, why each metric pairing is
required, calibration vs discrimination, disaggregated reporting, and the CLAIM 2024
reporting-fit map (/check-reporting owns the item audit).
Boundaries
model-validation (design) -> model-evaluation (this skill: per-case task-correct metrics + CIs)
-> analyze-stats (DeLong / NRI / IDI / decision curves, publication tables) -> make-figures
-> model-card (numbers + subgroup) -> write-paper + check-reporting
1---2name: model-evaluation3description: >4 Compute and report task-correct held-out metrics for a trained medical-imaging model — segmentation5 (Dice plus a boundary metric such as HD95 or NSD, per structure), classification (AUROC plus AUPRC and6 sensitivity/specificity with bootstrap CIs at the deployment prevalence), detection (FROC or mAP with7 a stated IoU criterion), interactive/promptable segmentation (the interaction-count, convergence,8 and per-case-time axes a static Dice omits), or generative/synthesis image evaluation (similarity plus9 the downstream-task efficacy similarity alone cannot establish) — plus calibration and subgroup slices.10 Emits a per-case results table that analyze-stats turns into publication tables, and gates the metric11 choice against Metrics Reloaded, CLAIM 2024, and Park et al. 2024 (no pixel accuracy for segmentation,12 no bare accuracy under imbalance, no static Dice for an interactive method, no similarity-only claim13 for a generative model). Numbers come only from executed code, never hand-typed.14triggers: model evaluation, held-out metrics, test set metrics, Dice, HD95, NSD, surface distance, Metrics Reloaded, AUROC, AUPRC, bootstrap CI, calibration, ECE, reliability diagram, subgroup analysis, slice metrics, mAP, FROC, segmentation metrics, detection metrics, evaluate predictions, interactive segmentation, promptable segmentation, SAM2, MedSAM2, nnInteractive, number of clicks, NoC, interactions-to-threshold, click budget, generative metrics, image synthesis, SSIM, PSNR, SNR, CNR, downstream task, multiclass classification, Obuchowski index, Harrell's C, time-dependent ROC15tools: Read, Write, Edit, Bash, Grep, Glob16model: inherit17metadata:18 source: https://skillmd.com/skills/aperivue/model-evaluation19---2021# Model-Evaluation Skill2223## Purpose2425This skill makes a medical-imaging model's **held-out evaluation task-correct and honest**: the right26metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It27emits a **per-case metric table** that the publication statistics build on, and gates the metric choice28against Metrics Reloaded (Maier-Hein & Reinke et al., *Nat Methods* 2024) and CLAIM 2024.2930It sits between `/model-validation` (which audits the split / design) and `/analyze-stats` (which owns31the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC,32ECE of a softmax head); `/analyze-stats` owns DeLong / NRI / IDI / decision curves / MRMC. Like33`/analyze-stats`, it **generates and executes** code on your predictions — numbers are never hand-typed.3435## When to use36- You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and37 subgroup slices, plus a per-case table for the manuscript statistics.3839## When NOT to use40- Auditing the validation design / leakage → `/model-validation`.41- DeLong / NRI / IDI / decision curves / MRMC reader study → `/analyze-stats`.42- Building / training the model → `/model-scaffold`; LLM / MLLM → `/mllm-eval`.43- Figure rendering → `/make-figures`.4445## Workflow4647### Phase 1 — Fix the analysis unit and the task48State the task (segmentation / classification / detection / interactive / generative) and the **analysis unit** the metric must49respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as50per-patient.5152### Phase 2 — Compute task-correct metrics53Generate evaluation code that computes, on the held-out predictions:54- **segmentation**: Dice/IoU **and** a boundary metric (HD95 / NSD), **per structure** not only a global55 mean, with bootstrap 95% CIs.56- **classification**: **AUROC and AUPRC** with bootstrap CIs, sensitivity/specificity, and PPV/NPV **at57 the deployment prevalence** (not a balanced set).58- **detection**: **FROC / mAP** with the **IoU match criterion stated**.59- **interactive / promptable segmentation** (SAM2 / MedSAM2 / nnInteractive): the segmentation60 metrics above **plus** the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a61 target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time62 (see the metric guide; the study design is in `/design-study` + `/model-validation`).63- **generative / synthesis** (image generation or modification): full-reference similarity64 (MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), **plus a65 downstream-task evaluation** — image quality is not clinical utility (Park et al., *Radiol Med* 2024).66 For **multiclass** classification, state the aggregation scheme (one-vs-rest / macro / micro /67 pairwise / Obuchowski); **time-to-event** discrimination (Harrell's C, time-dependent ROC) is handed68 to `/analyze-stats`.69Add **calibration** (reliability diagram / ECE) and **subgroup** slices (the Model Card Factors).70See `${CLAUDE_SKILL_DIR}/references/metric_guide.md`. Emit a **per-case CSV** for `/analyze-stats`.7172### Phase 3 — Gate the metric choice (deterministic)73```bash74python3 ${CLAUDE_SKILL_DIR}/scripts/check_metric_reporting.py \75 --report results.md --task segmentation|classification|detection|interactive|generative --strict76```77`PIXEL_ACCURACY_SEG` / `NO_BOUNDARY_METRIC` / `ACCURACY_ONLY` / `DETECTION_METRIC_MISSING` must be zero.7879### Phase 4 — Hand off80The per-case table → `/analyze-stats` (DeLong / NRI / IDI / decision curves, publication tables);81figures → `/make-figures`; the numbers + subgroup performance → `/model-card`; Methods/Results →82`/write-paper`; compliance → `/check-reporting`.8384## Anti-Hallucination8586- **Never fabricate a metric value.** Every number comes from executed code on the supplied predictions;87 if predictions or ground truth are missing, say so and stop — do not invent a result.88- **Never report pixel/voxel accuracy for segmentation or bare accuracy under imbalance** — the gate89 flags these; report Dice + a boundary metric, or AUROC + AUPRC with CIs.90- **Never report a per-lesion metric as if it were per-patient** — respect the analysis unit.91- If a metric definition or its CI method is uncertain, flag `[VERIFY]` and ask.9293## Deterministic gate94`scripts/check_metric_reporting.py` — flags a task-metric mismatch / missing uncertainty (stdlib,95network-free). Reproducible challenge: `bash ${CLAUDE_SKILL_DIR}/scripts/metric_reporting_challenge/verify.sh`.9697## Reference Files9899Load on demand (keep SKILL.md short):100- `${CLAUDE_SKILL_DIR}/references/metric_guide.md` — operational checklist: the task-correct metric101 per task (segmentation Dice + HD95/NSD per structure; classification AUROC + AUPRC + sens/spec at102 deployment prevalence; detection FROC/mAP with a stated IoU), plus calibration, subgroup slices,103 run-variance, and the per-case CSV hand-off.104- `${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md` — the standards grounding behind105 those choices: the Metrics Reloaded task-fingerprint principle, why each metric pairing is106 required, calibration vs discrimination, disaggregated reporting, and the CLAIM 2024107 reporting-fit map (`/check-reporting` owns the item audit).108109## Boundaries110111```112model-validation (design) -> model-evaluation (this skill: per-case task-correct metrics + CIs)113 -> analyze-stats (DeLong / NRI / IDI / decision curves, publication tables) -> make-figures114 -> model-card (numbers + subgroup) -> write-paper + check-reporting115```