Medconceptsqaeval

medconceptsqaeval

MedConceptsQA: Open Source Medical Concepts QA Benchmark — Ben Shoham et al. (2024) (arXiv:2405.07348, 2024)

What this evaluates

Evaluates large language models' ability to reason about and identify medical concepts (diagnoses, procedures, drugs) across different semantic hierarchies and difficulty levels.

Datasets

• MedConceptsQA — total 250; splits: test (250)

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly predicted answers out of total questions. Calculated as correct predictions divided by total predictions.

Input / output format

Input: A multiple-choice question about a medical concept (diagnosis, procedure, or drug) with randomized correct answer positions. For few-shot evaluation, 4 example question-answer pairs are prepended to the prompt.

Output: The model's selected answer choice (e.g., the medical code or concept name).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Evaluations are run on a randomly sampled subset of 250 Q&As per vocabulary/difficulty/shots combination, not the full benchmark.
• Results are averaged over three independent runs with 95% confidence intervals reported, rather than a single deterministic run.
• Correct answer positions are randomized during dataset creation to prevent position bias, so models cannot rely on answer location.

Evidence (verbatim from paper)

We use accuracy as the evaluation metric because the datasets in the benchmark are balanced, as we selected the placement of the correct answer randomly during the creation of the benchmark. For each dataset in the benchmark, we conduct zero-shot and few-shot evaluations at each difficulty level (easy, medium, and hard). We repeat each evaluation for each model three times and calculate a 95% confidence interval. Due to the large amount of resources required by some of the models, especially GPT4, we limit each type of test (vocabulary and difficulty and shots) to 250 randomly sampled Q&As.

Citation

@misc{ben_shoham2024medconceptsqabenchmark,
  title={MedConceptsQA: Open Source Medical Concepts QA Benchmark},
  author={Ben Shoham et al. (2024)},
  year={2024},
  note={arXiv:2405.07348}
}

• arXiv: 2405.07348