Medarabiq Eval

medarabiq-eval

MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks — Abu Daoud et al. (2025) (arXiv:2505.03427, 2025)

What this evaluates

Evaluates large language models on Arabic medical reasoning and dialogue across multiple-choice, fill-in-the-blank, and open-ended Q&A tasks. It probes factual accuracy, domain-specific knowledge, and robustness to linguistic variations and injected biases in healthcare contexts.

Datasets

• MedArabiQ — total ?; splits: test (-1); repo https://github.com/nyuad-cai/MedArabiQ

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answers out of total instances. Calculated as the number of exact matches between model predictions and gold labels divided by the total number of instances.
• BERTScore — range: [0, 1]
  • Semantic similarity metric computed using contextual embeddings from BERT to compare model outputs against reference answers. Values range from 0 to 1, with higher scores indicating better semantic alignment.

Input / output format

Input: Arabic medical prompts including multiple-choice questions, fill-in-the-blank statements, patient-doctor dialogue initiations, and questions modified with grammatical errors or LLM-generated alterations.

Output: For closed tasks: selected option or filled text. For open-ended tasks: free-form Arabic text responses.

Scoring recipe

def evaluate_medarabiq(predictions, golds, task_type):
    if task_type in ['MCQ', 'Fill-in-the-Blank']:
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
        return correct / len(golds)
    else:
        # BERTScore computation using contextual embeddings
        p, r, f1 = bert_score.score(predictions, golds, lang='ar')
        return f1.mean().item()

Common pitfalls

• Using exact-match accuracy for open-ended Q&A tasks instead of BERTScore, which would severely penalize semantically correct but paraphrased answers.
• Assuming uniform performance across Arabic dialects; the benchmark is constructed from formal medical exams, so models may underperform on colloquial or non-standard medical terminology.
• Comparing closed-task accuracy directly with open-task BERTScore without normalization, as the evaluation criteria and scales differ fundamentally.

Evidence (verbatim from paper)

We present the results in terms of accuracy and BERTScore, depending on task, and show best results in bold per row.

Citation

@misc{abudaoud2025medarabiq,
  title={MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks},
  author={Abu Daoud et al. (2025)},
  year={2025},
  note={arXiv:2505.03427}
}

• arXiv: 2505.03427