Caremedeval Eval

caremedeval-eval

CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field — Bonzi et al. (2025) (arXiv:2511.03441, 2025)

What this evaluates

This benchmark evaluates large language models' ability to perform critical appraisal and methodological reasoning on biomedical scientific articles. It probes whether models can correctly identify study design flaws, statistical limitations, and biases by answering multiple-choice questions derived from authentic French medical exams. The evaluation specifically measures both exact correctness and partial reasoning accuracy under varying context conditions.

Datasets

• CareMedEval — total 534; splits: test (534); repo https://github.com/bonzid/CareMedEval

Metrics

• Exact Match Ratio (EMR) (primary) — range: [0, 1]
  • Measures the proportion of questions for which the predicted set of answers exactly matches the gold standard answer set.
• F1-score — range: [0, 1]
  • The harmonic mean of precision and recall, computed between the predicted and gold answer sets.
• Hamming score — range: [0, 1]
  • Evaluates the proportion of correctly predicted labels (answer options) over the total number of possible labels, averaged over all questions.
• LCA score — range: [0, 1]
  • Custom exam-style grading: perfect match yields 1 point, one mismatch 0.5, two mismatches 0.25, and more than two mismatches or no response 0. Missing a required answer or including an unacceptable answer automatically forces a score of 0. Averaged over all questions.

Input / output format

Input: A French instruction-style prompt framing the model as a medical professional, containing a question, answer choices, and optionally the full article text, abstract only, or no article content depending on the evaluation scenario.

Output: A predicted set of answer options (multiple-choice selections) matching the specified output format in the prompt.

Scoring recipe

def compute_metrics(predictions, golds, required=None, unacceptable=None):
    emr = sum(1 for p, g in zip(predictions, golds) if set(p) == set(g)) / len(golds)
    lca_scores = []
    for p, g in zip(predictions, golds):
        p_set, g_set = set(p), set(g)
        if required and not required.issubset(p_set):
            lca_scores.append(0.0)
            continue
        if unacceptable and unacceptable.intersection(p_set):
            lca_scores.append(0.0)
            continue
        diff = len(g_set.symmetric_difference(p_set))
        lca_scores.append({0: 1.0, 1: 0.5, 2: 0.25}.get(diff, 0.0))
    lca = sum(lca_scores) / len(lca_scores)
    return {'EMR': emr, 'LCA': lca}

Common pitfalls

• The LCA score enforces strict constraints where missing a required answer or including an unacceptable answer automatically forces a score of 0, which differs from standard MCQA grading.
• Evaluation results vary significantly across input scenarios (full text, abstract only, or no text), so metrics must be reported per context condition rather than aggregated blindly.
• The dataset and prompts are in French, requiring models to handle domain-specific medical terminology and language correctly, which is often overlooked in cross-lingual evaluations.

Evidence (verbatim from paper)

Exact Match Ratio (EMR) measures the proportion of questions for which the predicted set of answers exactly matches the gold standard. F1-score is the harmonic mean of precision and recall, computed between predicted and gold answer sets. Hamming score evaluates the proportion of correctly predicted labels (answer options) over the total number of possible labels, averaged over all questions. LCA score is a custom metric inspired by the grading system used in the original LCA exam from which our dataset is derived.

Citation

@misc{bonzi2025caremedeval,
  title={CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field},
  author={Bonzi et al. (2025)},
  year={2025},
  note={arXiv:2511.03441}
}

• arXiv: 2511.03441