Emsqa Eval

emsqa-eval

Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering — Ge et al. (2025) (arXiv:2511.10900, 2025)

What this evaluates

Evaluates large language models and retrieval-augmented generation systems on emergency medical services (EMS) multiple-choice questions across different clinical subject areas and certification levels. It probes the models' ability to apply domain-specific expertise and reasoning to answer standardized medical certification questions.

Datasets

• EMSQA — total ?; splits: Public (-1), Private (-1)

Metrics

• exact-match accuracy (Acc) (primary) — range: percent
  • Percentage of samples where the model's predicted answer exactly matches the ground-truth answer.
• sample-based F1 — range: percent
  • F1 score computed per sample (treating answers as sets of labels due to multiple correct answers per question), then averaged across all samples.

Input / output format

Input: Multiple-choice medical question, optionally accompanied by retrieved clinical documents (KB) and patient records (PR) for RAG methods, or provided with few-shot examples/prompting instructions.

Output: Predicted answer(s) to the multiple-choice question (single or multiple correct options).

Scoring recipe

def score(predictions, golds):
    exact_matches = 0
    f1_scores = []
    for pred, gold in zip(predictions, golds):
        pred_set = set(pred)
        gold_set = set(gold)
        if pred_set == gold_set:
            exact_matches += 1
        tp = len(pred_set & gold_set)
        fp = len(pred_set - gold_set)
        fn = len(gold_set - pred_set)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
        f1_scores.append(f1)
    acc = exact_matches / len(predictions) * 100
    f1 = sum(f1_scores) / len(f1_scores) * 100
    return {'Acc': acc, 'F1': f1}

Common pitfalls

• Questions may have multiple correct answers, so exact-match requires predicting all correct options, not just one.
• The dataset is split into Public and Private subsets; performance should be reported separately to avoid data leakage or unfair comparison.
• RAG baselines use different retrieval corpora (KB, PR, or both), so comparing vanilla RAG to ExpertRAG requires matching the retrieval setup.

Evidence (verbatim from paper)

Since some questions in EMSQA have multiple correct answers, we report both exact-match accuracy (Acc) and sample-based F1*(Khashabi et al. [2018])*.

Citation

@misc{ge2025expertguided,
  title={Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering},
  author={Ge et al. (2025)},
  year={2025},
  note={arXiv:2511.10900}
}

• arXiv: 2511.10900