# Medarabiq Eval

> Evaluates large language models on Arabic medical reasoning and dialogue across multiple-choice, fill-in-the-blank, and open-ended Q&A tasks. It probes factual accuracy, domain-specific knowledge, and robustness to linguistic variations and injected biases in healthcare contexts. Use when the user wants to benchmark on MedArabiQ, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/medarabiq-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/medarabiq-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/medarabiq-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/medarabiq-eval

---


# medarabiq-eval

> MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks — Abu Daoud et al. (2025) (arXiv:2505.03427, 2025)

## What this evaluates

Evaluates large language models on Arabic medical reasoning and dialogue across multiple-choice, fill-in-the-blank, and open-ended Q&A tasks. It probes factual accuracy, domain-specific knowledge, and robustness to linguistic variations and injected biases in healthcare contexts.

## Datasets

- **MedArabiQ** — total ?; splits: test (-1); repo https://github.com/nyuad-cai/MedArabiQ

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted answers out of total instances. Calculated as the number of exact matches between model predictions and gold labels divided by the total number of instances.
- `BERTScore` — range: [0, 1]
  - Semantic similarity metric computed using contextual embeddings from BERT to compare model outputs against reference answers. Values range from 0 to 1, with higher scores indicating better semantic alignment.

## Input / output format

**Input**: Arabic medical prompts including multiple-choice questions, fill-in-the-blank statements, patient-doctor dialogue initiations, and questions modified with grammatical errors or LLM-generated alterations.

**Output**: For closed tasks: selected option or filled text. For open-ended tasks: free-form Arabic text responses.

## Scoring recipe

```python
def evaluate_medarabiq(predictions, golds, task_type):
    if task_type in ['MCQ', 'Fill-in-the-Blank']:
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
        return correct / len(golds)
    else:
        # BERTScore computation using contextual embeddings
        p, r, f1 = bert_score.score(predictions, golds, lang='ar')
        return f1.mean().item()
```

## Common pitfalls

- Using exact-match accuracy for open-ended Q&A tasks instead of BERTScore, which would severely penalize semantically correct but paraphrased answers.
- Assuming uniform performance across Arabic dialects; the benchmark is constructed from formal medical exams, so models may underperform on colloquial or non-standard medical terminology.
- Comparing closed-task accuracy directly with open-task BERTScore without normalization, as the evaluation criteria and scales differ fundamentally.

## Evidence (verbatim from paper)

> We present the results in terms of accuracy and BERTScore, depending on task, and show best results in bold per row.

## Citation

```bibtex
@misc{abudaoud2025medarabiq,
  title={MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks},
  author={Abu Daoud et al. (2025)},
  year={2025},
  note={arXiv:2505.03427}
}
```

- arXiv: 2505.03427