Clinical Note Understanding Eval

clinical-note-understanding-eval

Hierarchical Annotation for Building A Suite of Clinical Natural Language Processing Tasks: Progress Note Understanding — Gao et al. (2022) (arXiv:2204.03035, 2022)

What this evaluates

Evaluates NLP models on hierarchical clinical reasoning tasks, including SOAP section segmentation, diagnostic inference via assessment-plan relation labeling, and clinical summarization through problem/action plan extraction.

Datasets

• Clinical Progress Notes (MIMIC-III) — total 768; splits: train (608), dev (76), test (87)

Metrics

• Cohen's Kappa (primary) — range: [-1, 1]
  • Measures inter-annotator agreement for categorical labels while correcting for chance agreement. Computed on the AP relation labeling task across four classes (DIRECT, INDIRECT, NEITHER, NOT REL).

Input / output format

Input: Raw clinical progress note text.

Output: Hierarchical XML annotations containing section tags (e.g., SOAP attributes), assessment-plan relations (DIRECT, INDIRECT, NEITHER, NOT REL), and problem/action plan lists.

Scoring recipe

def compute_cohens_kappa(labels1, labels2, classes):
    N = len(labels1)
    observed = sum(1 for a, b in zip(labels1, labels2) if a == b) / N
    expected = sum((labels1.count(c) / N) * (labels2.count(c) / N) for c in classes)
    if expected == 1.0:
        return 0.0
    return (observed - expected) / (1 - expected)

Common pitfalls

• The dataset focuses on hierarchical clinical reasoning rather than simple entity extraction, requiring models to understand temporal and causal links between assessment and plan sections.
• Inter-annotator agreement is moderate (0.74) due to the ambiguity between INDIRECT and NEITHER relations, indicating inherent subjectivity in clinical reasoning tasks that models must navigate.

Evidence (verbatim from paper)

We measured Cohen's Kappa on the AP relation labeling task, as it was deemed the most difficult by the annotators because it was the only task that required clinical reasoning and medical knowledge. The two annotators achieved a Cohen's Kappa of 0.74 on 10 randomly sampled notes, which represented good quality given the complexity of the task.

Citation

@misc{gao2022hierarchical,
  title={Hierarchical Annotation for Building A Suite of Clinical Natural Language Processing Tasks: Progress Note Understanding},
  author={Gao et al. (2022)},
  year={2022},
  note={arXiv:2204.03035}
}

• arXiv: 2204.03035