qhjqhj00 medical-emr-extraction-eval
EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records — Zhao et al. (2025) (arXiv:2504.16448, 2025)
What this evaluates
Probes a model's ability to extract structured clinical information from unstructured physician-patient consultation dialogues. It evaluates how accurately the model maps conversational text into predefined medical record fields such as chief complaint, diagnosis, and treatment recommendations.
Datasets
- EMRModel Dataset — total 8665; splits: train (7329), val (800), test (536)
Metrics
weighted average F1 score(primary) — range: [0, 1]- Calculates the F1 score for each target field, then computes a weighted average where weights are the character counts of each field in the ground truth. The formula is F1_overall = sum(w_f * F1_f) / sum(w_f) for total weight > 0, else 0.
Input / output format
Input: Unstructured text of physician-patient consultation dialogues (transcribed from audio recordings).
Output: Structured medical record template containing predefined fields (e.g., age, gender, chief complaint, past medical history, preliminary diagnosis, treatment recommendations).
Scoring recipe
def compute_weighted_f1(preds, golds, fields):
total_weighted_f1 = 0.0
total_weight = 0.0
for f in fields:
pred_val = preds.get(f, "")
gold_val = golds.get(f, "")
f1_val = f1_score(gold_val, pred_val)
w_f = len(gold_val)
total_weighted_f1 += w_f * f1_val
total_weight += w_f
return total_weighted_f1 / total_weight if total_weight > 0 else 0.0
Common pitfalls
- Using a simple arithmetic mean across fields instead of the character-weighted average, which misrepresents the importance of longer clinical fields.
- Treating missing or empty model outputs inconsistently; the protocol explicitly states missing fields should be treated as empty strings for F1 calculation.
- Ignoring the sample standard deviation of F1 scores, which is required to assess performance stability across different test samples.
Evidence (verbatim from paper)
To comprehensively and accurately evaluate the performance of the LLM on the task of generating structured medical records from unstructured text, this study employs the weighted average F1 score as the primary evaluation metric. To assess the model’s performance stability across different samples, we further calculate the sample standard deviation of the weighted average F1 scores obtained for each individual sample. This standard deviation reflects the variability in the model’s performance when processing different test samples. A lower standard deviation indicates more stable model performance.
Citation
@misc{zhao2025emrmodel,
title={EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records},
author={Zhao et al. (2025)},
year={2025},
note={arXiv:2504.16448}
}
- arXiv: 2504.16448