# Medexqa Eval

> Evaluates medical language models on multiple-choice question answering and the generation of clinically relevant explanations. It probes the model's ability to perform clinical reasoning, avoid hallucinations, and produce coherent, accurate rationales aligned with medical domain knowledge. Use when the user wants to benchmark on MedExQA, or asks about evaluating this task. Reports Classification Accuracy.

- Skill: `qhjqhj00/medexqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/medexqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/medexqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/medexqa-eval

---


# medexqa-eval

> MedExQA: Medical Question Answering Benchmark with Multiple Explanations — Kim et al. (2024) (arXiv:2406.06331, 2024)

## What this evaluates

Evaluates medical language models on multiple-choice question answering and the generation of clinically relevant explanations. It probes the model's ability to perform clinical reasoning, avoid hallucinations, and produce coherent, accurate rationales aligned with medical domain knowledge.

## Datasets

- **MedExQA** — total ?; splits: test (-1), dev (-1)

## Metrics

- `Classification Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice questions. For generative models, the next token with the highest logit is selected, or string matching is used to find the answer choice/letter in the generated text. Score is 1 if match, 0 otherwise.
- `BLEU` — range: [0, 1]
  - Geometric mean of precision scores of generated explanations compared to reference explanations based on n-gram matches.
- `ROUGE-L` — range: [0, 1]
  - Measures similarity based on the longest common subsequence between generated and reference explanations, combining precision and recall.
- `METEOR` — range: [0, 1]
  - Considers semantic similarity and lexical variations using WordNet alignment between generated and reference texts.
- `BERTScore` — range: [0, 1]
  - Uses contextual embeddings (SciBERT) to capture semantic nuances between generated and reference explanations.
- `Human Evaluation Score` — range: [0, 1]
  - Annotators assign 0 for incorrect/irrelevant/missing explanations, 0.5 for correct answer but incorrect/incomplete explanation, and 1.0 for both correct answer and explanation.

## Input / output format

**Input**: Multiple-choice medical questions with four options, plus reference explanations for generation tasks.

**Output**: For classification: the predicted answer choice (letter or exact phrase). For explanation generation: a textual rationale explaining the correct answer.

## Scoring recipe

```python
def evaluate_medexqa(predictions, golds):
    acc_scores = []
    lex_scores = []
    for pred, gold in zip(predictions, golds):
        is_correct = (pred['answer'] == gold['answer']) or (gold['answer'] in pred['text'])
        acc_scores.append(1.0 if is_correct else 0.0)
        lex_scores.append(bertscore_f1(pred['explanation'], gold['explanation']))
    combined = [l if a == 1.0 else 0.0 for a, l in zip(acc_scores, lex_scores)]
    return {
        'accuracy': sum(acc_scores) / len(acc_scores),
        'bertscore': sum(lex_scores) / len(lex_scores),
        'combined': sum(combined) / len(combined)
    }
```

## Common pitfalls

- Logit-based accuracy cannot be computed for closed-source APIs (e.g., GPT-3.5/4), requiring fallback to string matching which may miss paraphrased answers.
- Reporting lexical metrics (BLEU/ROUGE/BERTScore) without filtering out incorrect answers inflates performance, as the protocol mandates a score of 0 for wrong answers.
- Human evaluation was only performed on a small development subset (5 samples per specialty), so automated metrics are the primary reported benchmarks.

## Evidence (verbatim from paper)

> The quality of generated explanations is further assessed using a combination of general lexical metrics. BLEU measures the geometric mean of precision scores of the generated explanations compared to reference explanations based on n-gram matches. ROUGE assesses the similarity between generated and reference explanations, with ROUGE-L, providing a score that combines precision and recall based on the longest common subsequence. METEOR considers the semantic similarity and lexical variations with WordNet. BERTScore uses contextual embeddings, scibert embedding for our work, to capture nuances in the semantics of the explanations. We propose an enhanced methodology for evaluating models’ understanding of medical domain knowledge by incorporating classification accuracy based on string matches into calculating these metrics. We assign a score of 0 to responses with incorrect answers based on string-matching classification results.

## Citation

```bibtex
@misc{kim2024medexqa,
  title={MedExQA: Medical Question Answering Benchmark with Multiple Explanations},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2406.06331}
}
```

- arXiv: 2406.06331