Emrqa Eval

emrqa-eval

emrQA: A Large Corpus for Question Answering on Electronic Medical Records — Pampari et al. (2018) (arXiv:1809.00732, 2018)

What this evaluates

Evaluates a model's ability to map clinical questions to structured logical forms and to extract precise answer spans or predict answer classes from unstructured electronic medical records. It probes complex clinical reasoning, including temporal, arithmetic, and multi-sentence contextual understanding.

Datasets

• emrQA — total ?; splits: emrQL-1 train (1000000), emrQL-1 test (253000), emrQL-2 train (1100000), emrQL-2 test (296000), QA train (47605), QA test (9966); repo https://github.com/panushri25/emrQA

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly predicted logical forms out of the total number of test instances.
• Exact Match (EM) (primary) — range: [0, 1]
  • Modified for EMRs: scores 1 if the predicted evidence span contains the answer entity or lies within ±20 characters of the ground truth evidence; otherwise 0. Averaged over the top 10 model predictions and the number of ground truth answers per question.
• F1 — range: [0, 1]
  • Bag-of-tokens F1 score measuring overlap between predicted and ground truth evidence spans. Averaged over the top 10 model predictions and the number of ground truth answers per question.

Input / output format

Input: Question text paired with the full clinical note (Electronic Medical Record) from which the answer must be derived.

Output: For Q-L mapping: a structured logical form. For Q-A mapping: an extracted evidence span from the clinical note or a predicted answer class.

Scoring recipe

em_scores = []
f1_scores = []
for q in questions:
    gt_answers = get_ground_truth_answers(q)
    top_preds = get_top_10_predictions(q)
    for pred in top_preds:
        for gt in gt_answers:
            if gt.entity in pred.text or abs(len(pred.text) - len(gt.text)) <= 20:
                em_scores.append(1.0)
            else:
                em_scores.append(0.0)
            tokens_pred = set(pred.text.split())
            tokens_gt = set(gt.text.split())
            intersection = len(tokens_pred & tokens_gt)
            denom = len(tokens_pred) + len(tokens_gt)
            f1_scores.append(2 * intersection / denom if denom > 0 else 0.0)
em_metric = sum(em_scores) / len(em_scores)
f1_metric = sum(f1_scores) / len(f1_scores)

Common pitfalls

• The EM metric uses a non-standard ±20 character tolerance instead of exact string matching.
• Scores are averaged over the top 10 predictions and multiple ground truth answers per question, which differs from standard single-answer QA benchmarks.
• The dataset provides two alternative train/test splits (emrQL-1 and emrQL-2) with different lexical/paraphrase distributions; results are not directly comparable across splits.

Evidence (verbatim from paper)

Wherever the answer entity in an evidence is explicitly known, EM checks if the answer entity is present within the evidence, otherwise it checks if the predicted evidence span lies within ±20 characters of the ground truth evidence. For F1 we construct a bag of tokens for each evidence string and measure the F1 score of the overlap between the two bags of tokens. Since there may be multiple evidence for a given question, we consider only the top 10 predictions and report an average of EM and F1 over ground truth number of answers.

Citation

@misc{pampari2018emrqa,
  title={emrQA: A Large Corpus for Question Answering on Electronic Medical Records},
  author={Pampari et al. (2018)},
  year={2018},
  note={arXiv:1809.00732}
}

• arXiv: 1809.00732