Soap Note Generation Eval

soap-note-generation-eval

Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization — Labrak et al. (2026) (arXiv:2604.06138, 2026)

What this evaluates

Evaluates the ability of audio and text models to generate clinically accurate, well-structured SOAP notes from long-form doctor-patient conversations. Probes long-context audio reasoning, fact-grounding, and clinical documentation quality.

Datasets

• Doctor-Patient SOAP Conversations — total 8800; splits: dev (-1), test (-1)

Metrics

• Faithfulness (primary) — range: other
  • LLM-as-a-judge scores claim support and contradictions on a 1–5 scale.
• Coverage — range: other
  • LLM-as-a-judge scores completeness of documentation relative to transcript evidence on a 1–5 scale.
• ROUGE-2 F1 — range: percent
  • Standard ROUGE-2 F1 score computed between generated and reference SOAP notes.
• ROUGE-3 F1 — range: percent
  • Standard ROUGE-3 F1 score computed between generated and reference SOAP notes.
• ROUGE-L F1 — range: percent
  • Standard ROUGE-L F1 score computed between generated and reference SOAP notes.
• Open Medical Concept F1 — range: percent
  • F1 score over overlap of medical concepts extracted via MeSH keyword matching and NER (en_core_sci_md) from reference and hypothesis notes.
• Unsupported claim rate — range: [0, 1]
  • Proportion of atomic claims in the generated note that lack grounding in the transcript.
• Contradiction rate — range: [0, 1]
  • Proportion of atomic claims in the generated note that contradict the transcript.

Input / output format

Input: Multi-speaker doctor-patient audio conversation (for end-to-end models) or speaker-attributed transcript (for cascaded models).

Output: SOAP note text containing History of Present Illness (HPI), Review of Systems (ROS), Assessment, and Plan sections.

Scoring recipe

def evaluate(generated_note, reference_note, transcript):
    claims = extract_atomic_claims(generated_note)
    faithfulness = 0
    for claim in claims:
        evidence = transcript.search(claim)
        if evidence.supports: faithfulness += 1
        elif evidence.contradicts: faithfulness -= 1
    faithfulness = scale_1_to_5(faithfulness / len(claims))
    rouge2 = compute_rouge_n(generated_note, reference_note, n=2)
    rouge3 = compute_rouge_n(generated_note, reference_note, n=3)
    rouge_l = compute_rouge_l(generated_note, reference_note)
    med_f1 = compute_f1(
        extract_medical_concepts(generated_note, method="MeSH+scispaCy"),
        extract_medical_concepts(reference_note, method="MeSH+scispaCy")
    )
    unsupported_rate = count_unsupported_claims(claims, transcript) / len(claims)
    contradiction_rate = count_contradicting_claims(claims, transcript) / len(claims)
    return faithfulness, rouge2, rouge3, rouge_l, med_f1, unsupported_rate, contradiction_rate

Common pitfalls

• Reference notes are LLM-generated rather than physician-authored, which may introduce systematic biases in both content and evaluation signals.
• Acoustic simulation uses low-WER synthetic audio (<3% for Whisper), so results may not generalize to real clinical recordings with high background noise or overlapping speech.
• End-to-end models are evaluated on raw audio while cascaded models use transcripts, making direct comparison sensitive to ASR error propagation and modality gaps.

Evidence (verbatim from paper)

We score SOAP notes along 12 dimensions. Four dimensions are scored on a 1–5 scale (5 = best): Faithfulness (claim support and contradictions), Structure (SOAP formatting and section placement), Coverage (completeness of documentation relative to transcript evidence), and Conciseness (redundancy and low-value content). We additionally evaluate two rates: unsupported claim rate and contradiction rate. In addition to the reference-free LLM judge, we evaluate generated notes against the reference using standard ROUGE F1 metrics (R-2, R-3, R-L) and an Open Medical Concept metric, which extracts medical concepts from both reference and hypothesis notes and computes F1 over their overlap.

Citation

@misc{labrak2026generatingsynthetic,
  title={Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization},
  author={Labrak et al. (2026)},
  year={2026},
  note={arXiv:2604.06138}
}

• arXiv: 2604.06138