Mimic Iii Clinical Fact Checking Eval

mimic-iii-clinical-fact-checking-eval

Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation — Musarrat Zeba et al. (2025) (arXiv:2512.16189, 2025)

What this evaluates

Evaluates the factual consistency and logical coherence of LLM-generated clinical discharge summaries against ground-truth Electronic Health Records (EHRs) at a granular propositional level. It measures how accurately a model's extracted propositions align with clinician-validated EHR facts.

Datasets

• MIMIC-III — total 26104; splits: train (20883), val (2610), test (2611)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for binary proposition classification (Supported vs. Not Supported). Calculated as 2 * (precision * recall) / (precision + recall).
• precision — range: [0, 1]
  • Ratio of correctly identified supported propositions to all propositions labeled as supported by the model.
• recall — range: [0, 1]
  • Ratio of correctly identified supported propositions to all actual supported propositions in the ground truth EHR.

Input / output format

Input: Paired clinical discharge summaries and corresponding Electronic Health Records (EHRs), with propositions extracted from both for comparison.

Output: Binary classification label per proposition ('Supported' or 'Not Supported') and generated clinical summary text.

Scoring recipe

def compute_metrics(gold_labels, pred_labels):
    tp = sum(1 for g, p in zip(gold_labels, pred_labels) if g == 'Supported' and p == 'Supported')
    fp = sum(1 for g, p in zip(gold_labels, pred_labels) if g == 'Not Supported' and p == 'Supported')
    fn = sum(1 for g, p in zip(gold_labels, pred_labels) if g == 'Supported' and p == 'Not Supported')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Proposition-level annotation requires significant clinician time, limiting the fact-checking evaluation to a small stratified subset (104 records) rather than the full 26k dataset.
• Binary 'Supported/Not Supported' labels may oversimplify nuanced clinical inconsistencies or rare condition representations that clinicians noted during human evaluation.

Evidence (verbatim from paper)

We evaluated the correctness of each proposition of our fact-checking module using precision, recall, F1-score, and confusion matrix analysis. These metrics allow us to understand how well the generated summaries align with their respective EHR data.

Citation

@misc{zeba2025mitigating,
  title={Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation},
  author={Musarrat Zeba et al. (2025)},
  year={2025},
  note={arXiv:2512.16189}
}

• arXiv: 2512.16189