Uncertainty-Imaging Skill
Purpose
A medical-imaging model framed for deployment must say more than "class 1, 0.87". It needs a
calibrated uncertainty on each case, an out-of-distribution (OOD) guard validated on data known
to be out-of-distribution, and — if it abstains — a pre-specified operating point. The failures are
predictable and reviewer-visible: a clinical-use claim built on point predictions, conformal intervals
quoted without ever measuring their coverage, an "OOD detector" evaluated only on in-distribution data,
a deep ensemble whose members share a seed, and uncertainty validated only in-distribution when
deployment sees scanner/site/case-mix shift. This skill designs that layer and audits an existing one
(Gal 2016; Lakshminarayanan 2017; Angelopoulos & Bates; Ovadia 2019; DECIDE-AI).
It is the deployment-safety companion in the model-engineering lane: /model-evaluation computes the
held-out metrics and calibration, and uncertainty-imaging covers the uncertainty / OOD / abstention
machinery a deployment claim rests on. It integrates MAPIE (conformal), captum, and pretrained OOD
scorers; it does not reimplement them and never runs a model on real patient data.
When to use
- Your model is framed for clinical use / deployment and a reviewer will ask "what does it do when it is
unsure, or off-distribution?"
- You report conformal / MC-dropout / ensemble uncertainty and want the coverage, independence, and
shift checks right before submission.
- You want to audit an existing uncertainty/OOD section for the failure modes below.
When NOT to use
- Held-out discrimination / calibration metrics of the point predictor →
/model-evaluation then
/analyze-stats.
- Training-repo scaffolding / the split →
/model-scaffold (+ /model-validation).
- Interpretability / saliency of a trained network →
/explainability.
- Classical-ML calibration of a tabular model →
/radiomics-ml + /analyze-stats.
- Reimplementing MAPIE / an OOD library → out of scope (this skill wires and audits them).
The failure modes (what the gate enforces)
- Point predictions under a deployment claim. A clinical-use claim with no uncertainty method at
all — add MC-dropout, a deep ensemble, conformal prediction, or a Bayesian estimate.
- Conformal without coverage validation. Conformal's guarantee holds under exchangeability, which
can fail on clinical data — measure achieved coverage on a held-out calibration/test set.
- OOD claim with no held-out OOD set. An OOD detector's operating point and AUROC are unmeasured
until you evaluate on data known to be out-of-distribution (different scanner / site / pathology).
- Non-independent ensemble. A deep ensemble whose members share a seed/init (or has < 2 members)
underestimates epistemic uncertainty.
- MC-dropout with dropout off at inference. Dropout must stay active during sampling; off, every
pass is identical and the estimate collapses to a point prediction.
- Selective prediction without a target. Abstention chosen post hoc inflates accuracy-at-coverage;
pre-specify the coverage / risk operating point.
- No calibration under shift. Uncertainty evaluated in-distribution only; deployment uncertainty
degrades under shift, so report it on shifted / external data.
Workflow
Phase 1 — Choose the uncertainty method (integrate, don't reimplement)
- Conformal prediction (MAPIE) — distribution-free prediction sets/intervals at a nominal coverage;
the strongest default when a calibration set is available. Validate empirical coverage.
- Deep ensembles (Lakshminarayanan 2017) — train K independent members (distinct seeds/inits); the
best-quality epistemic uncertainty, at K× cost.
- MC-dropout (Gal 2016) — keep dropout active at inference and sample T passes; cheap, weaker.
- Bayesian / Laplace — a last-layer Laplace approximation is a light option.
See
references/uncertainty_guide.md.
Phase 2 — Add the OOD guard and the abstention rule
- OOD detection — an energy score, Mahalanobis distance on features, ODIN, or max-softmax; evaluate
on a held-out OOD set (different scanner/site/pathology) and report detection AUROC + the operating
point.
- Selective prediction — abstain below a confidence/uncertainty threshold set to a pre-specified
target coverage or risk; report the risk–coverage curve.
Phase 3 — Stress it under shift
Report calibration / coverage on shifted or external data, not in-distribution only (Ovadia 2019).
Phase 4 — Emit the uncertainty manifest
{
"task": "classification",
"deployment_claim": true,
"uncertainty_method": "conformal",
"coverage_target": 0.90,
"coverage_validated": true,
"ood_method": "mahalanobis",
"ood_heldout_set": "external-ood-cohort",
"selective_prediction": true,
"selective_target": 0.95,
"calibration_under_shift": true
}
Phase 5 — Gate the spec (deterministic)
python3 scripts/check_uncertainty_reporting.py --manifest uncertainty_manifest.json --strict
Verdicts: POINT_PREDICTION_NO_UNCERTAINTY, CONFORMAL_NO_COVERAGE_VALIDATION, OOD_NO_HELDOUT_SET
(Major); ENSEMBLE_NOT_INDEPENDENT, MCDROPOUT_DISABLED_AT_INFERENCE, SELECTIVE_NO_TARGET,
NO_CALIBRATION_UNDER_SHIFT (Minor). Audits the declared spec at design/report time; it complements
/model-evaluation's executed calibration/subgroup metrics.
Integration
/model-evaluation — the point predictor's held-out metrics + calibration this layer sits on top of.
/analyze-stats — calibration curve / risk–coverage plotting for the report.
/check-reporting — TRIPOD+AI / DECIDE-AI deployment-monitoring items.
/model-validation — the DECIDE-AI monitoring seam (the deployment-time counterpart of the split
audit).
Anti-Hallucination
- Never fabricate coverage, OOD AUROC, or calibration numbers. Every value in the manifest and every
reported number comes from the researcher's executed code — never invented. This skill designs and
audits the uncertainty spec; it does not run a model on real patient data.
- Never report conformal coverage as guaranteed without measuring it. Exchangeability can fail on
clinical data (
CONFORMAL_NO_COVERAGE_VALIDATION).
- Never report an uncertainty/OOD audit "pass" without running
check_uncertainty_reporting.py. The
verdict is reproduced deterministically, never asserted from prose.
- Integrate, don't reimplement. Reference MAPIE / captum / OOD scorers; do not write a new conformal
or OOD library or claim results for one.
Reproducible challenge
scripts/check_uncertainty_reporting_challenge/ ships a synthetic weak/strong uncertainty-manifest pair
with a network-free verify.sh wired into the skill's validation commands.
1---2name: uncertainty-imaging3description: >4 Design or audit the uncertainty-quantification, out-of-distribution (OOD) detection, and5 selective-prediction layer of a medical-imaging model framed for deployment — so a clinical-use claim6 carries calibrated per-case uncertainty (MC-dropout / deep ensemble / conformal / Bayesian), an OOD7 guard validated on a held-out OOD set, an abstention rule at a pre-specified operating point, and8 uncertainty checked under distribution shift. Emits an uncertainty manifest and a deterministic gate9 that flags a deployment claim built on point predictions, conformal intervals with unmeasured coverage,10 and an OOD claim with no held-out OOD data. Integrates MAPIE / captum / pretrained OOD scorers; it does11 not reimplement them and never runs a model on real patient data.12triggers: uncertainty, uncertainty quantification, UQ, epistemic, aleatoric, MC-dropout, monte carlo dropout, deep ensemble, conformal prediction, split conformal, prediction interval, coverage, calibration under shift, out-of-distribution, OOD detection, distribution shift, Mahalanobis, energy score, ODIN, selective prediction, abstention, reject option, deployment safety, DECIDE-AI, predictive uncertainty13tools: Read, Write, Edit, Bash, Grep, Glob14model: inherit15metadata:16 source: https://skillmd.com/skills/aperivue/uncertainty-imaging17---1819# Uncertainty-Imaging Skill2021## Purpose2223A medical-imaging model framed for **deployment** must say more than "class 1, 0.87". It needs a24**calibrated uncertainty** on each case, an **out-of-distribution (OOD) guard** validated on data known25to be out-of-distribution, and — if it abstains — a **pre-specified operating point**. The failures are26predictable and reviewer-visible: a clinical-use claim built on point predictions, conformal intervals27quoted without ever measuring their coverage, an "OOD detector" evaluated only on in-distribution data,28a deep ensemble whose members share a seed, and uncertainty validated only in-distribution when29deployment sees scanner/site/case-mix shift. This skill designs that layer and audits an existing one30(Gal 2016; Lakshminarayanan 2017; Angelopoulos & Bates; Ovadia 2019; DECIDE-AI).3132It is the deployment-safety companion in the model-engineering lane: `/model-evaluation` computes the33held-out metrics and calibration, and **uncertainty-imaging** covers the uncertainty / OOD / abstention34machinery a deployment claim rests on. It **integrates** MAPIE (conformal), captum, and pretrained OOD35scorers; it does not reimplement them and never runs a model on real patient data.3637## When to use38- Your model is framed for clinical use / deployment and a reviewer will ask "what does it do when it is39 unsure, or off-distribution?"40- You report conformal / MC-dropout / ensemble uncertainty and want the coverage, independence, and41 shift checks right before submission.42- You want to audit an existing uncertainty/OOD section for the failure modes below.4344## When NOT to use45- Held-out discrimination / calibration metrics of the point predictor → `/model-evaluation` then46 `/analyze-stats`.47- Training-repo scaffolding / the split → `/model-scaffold` (+ `/model-validation`).48- Interpretability / saliency of a trained network → `/explainability`.49- Classical-ML calibration of a tabular model → `/radiomics-ml` + `/analyze-stats`.50- Reimplementing MAPIE / an OOD library → out of scope (this skill wires and audits them).5152## The failure modes (what the gate enforces)531. **Point predictions under a deployment claim.** A clinical-use claim with no uncertainty method at54 all — add MC-dropout, a deep ensemble, conformal prediction, or a Bayesian estimate.552. **Conformal without coverage validation.** Conformal's guarantee holds under exchangeability, which56 can fail on clinical data — measure achieved coverage on a held-out calibration/test set.573. **OOD claim with no held-out OOD set.** An OOD detector's operating point and AUROC are unmeasured58 until you evaluate on data known to be out-of-distribution (different scanner / site / pathology).594. **Non-independent ensemble.** A deep ensemble whose members share a seed/init (or has < 2 members)60 underestimates epistemic uncertainty.615. **MC-dropout with dropout off at inference.** Dropout must stay active during sampling; off, every62 pass is identical and the estimate collapses to a point prediction.636. **Selective prediction without a target.** Abstention chosen post hoc inflates accuracy-at-coverage;64 pre-specify the coverage / risk operating point.657. **No calibration under shift.** Uncertainty evaluated in-distribution only; deployment uncertainty66 degrades under shift, so report it on shifted / external data.6768## Workflow6970### Phase 1 — Choose the uncertainty method (integrate, don't reimplement)71- **Conformal prediction** (MAPIE) — distribution-free prediction sets/intervals at a nominal coverage;72 the strongest default when a calibration set is available. Validate empirical coverage.73- **Deep ensembles** (Lakshminarayanan 2017) — train K independent members (distinct seeds/inits); the74 best-quality epistemic uncertainty, at K× cost.75- **MC-dropout** (Gal 2016) — keep dropout active at inference and sample T passes; cheap, weaker.76- **Bayesian / Laplace** — a last-layer Laplace approximation is a light option.77See `references/uncertainty_guide.md`.7879### Phase 2 — Add the OOD guard and the abstention rule80- **OOD detection** — an energy score, Mahalanobis distance on features, ODIN, or max-softmax; **evaluate81 on a held-out OOD set** (different scanner/site/pathology) and report detection AUROC + the operating82 point.83- **Selective prediction** — abstain below a confidence/uncertainty threshold set to a **pre-specified**84 target coverage or risk; report the risk–coverage curve.8586### Phase 3 — Stress it under shift87Report calibration / coverage on **shifted or external** data, not in-distribution only (Ovadia 2019).8889### Phase 4 — Emit the uncertainty manifest90```json91{92 "task": "classification",93 "deployment_claim": true,94 "uncertainty_method": "conformal",95 "coverage_target": 0.90,96 "coverage_validated": true,97 "ood_method": "mahalanobis",98 "ood_heldout_set": "external-ood-cohort",99 "selective_prediction": true,100 "selective_target": 0.95,101 "calibration_under_shift": true102}103```104105### Phase 5 — Gate the spec (deterministic)106```bash107python3 scripts/check_uncertainty_reporting.py --manifest uncertainty_manifest.json --strict108```109Verdicts: `POINT_PREDICTION_NO_UNCERTAINTY`, `CONFORMAL_NO_COVERAGE_VALIDATION`, `OOD_NO_HELDOUT_SET`110(Major); `ENSEMBLE_NOT_INDEPENDENT`, `MCDROPOUT_DISABLED_AT_INFERENCE`, `SELECTIVE_NO_TARGET`,111`NO_CALIBRATION_UNDER_SHIFT` (Minor). Audits the declared spec at design/report time; it complements112`/model-evaluation`'s executed calibration/subgroup metrics.113114## Integration115- **`/model-evaluation`** — the point predictor's held-out metrics + calibration this layer sits on top of.116- **`/analyze-stats`** — calibration curve / risk–coverage plotting for the report.117- **`/check-reporting`** — TRIPOD+AI / DECIDE-AI deployment-monitoring items.118- **`/model-validation`** — the DECIDE-AI monitoring seam (the deployment-time counterpart of the split119 audit).120121## Anti-Hallucination122123- **Never fabricate coverage, OOD AUROC, or calibration numbers.** Every value in the manifest and every124 reported number comes from the researcher's executed code — never invented. This skill designs and125 audits the uncertainty spec; it does not run a model on real patient data.126- **Never report conformal coverage as guaranteed without measuring it.** Exchangeability can fail on127 clinical data (`CONFORMAL_NO_COVERAGE_VALIDATION`).128- **Never report an uncertainty/OOD audit "pass" without running `check_uncertainty_reporting.py`.** The129 verdict is reproduced deterministically, never asserted from prose.130- **Integrate, don't reimplement.** Reference MAPIE / captum / OOD scorers; do not write a new conformal131 or OOD library or claim results for one.132133## Reproducible challenge134`scripts/check_uncertainty_reporting_challenge/` ships a synthetic weak/strong uncertainty-manifest pair135with a network-free `verify.sh` wired into the skill's validation commands.