balsam-eval
BALSAM: A Platform for Benchmarking Arabic Large Language Models — Al-Matham et al. (2025) (arXiv:2507.22603, 2025)
What this evaluates
Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning.
Datasets
Metrics
LLM as a judge (primary) — range: [0, 3]
- An LLM (Gemini 2.5 Flash) first extracts a concise answer from the model's verbose output, then rates it on a 0–3 scale mirroring human evaluation guidelines.
Input / output format
Input: Arabic language prompts/tasks spanning 14 categories (e.g., Creative Writing, QA, Translation, Logic, Program Execution).
Output: Model-generated text responses.
Scoring recipe
def score_llm_judge(predictions, gold):
extracted = [llm_extract(p, prompt=EXTRACTION_PROMPT) for p in predictions]
scores = [llm_rate(e, g, prompt=SCORING_PROMPT, scale=[0,1,2,3]) for e, g in zip(extracted, gold)]
return sum(scores) / len(scores)
Common pitfalls
- Traditional n-gram metrics (BLEU/ROUGE) heavily penalize verbose outputs and fail to handle Arabic's complex morphology, leading to artificially low scores for longer, correct answers.
- Model size is not a reliable predictor of Arabic capability; tokenization quality, Arabic training data volume, and Arabic-specific SFT are more critical factors.
Evidence (verbatim from paper)
Table [5] shows the correlation of human evaluation with ROUGE-LSum, BLEU, and BERTScore (with and without extraction of answers using an LLM) and LLM as a judge. We make the following observations: ... LLM as a judge was highly correlated with human judgments for all categories, with values ranging between 0.824 and 0.977.
Citation
@misc{almatham2025balsam,
title={BALSAM: A Platform for Benchmarking Arabic Large Language Models},
author={Al-Matham et al. (2025)},
year={2025},
note={arXiv:2507.22603}
}
1---2name: balsam-eval3description: Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning. Use when the user wants to benchmark on BALSAM, or asks about evaluating this task. Reports LLM as a judge.4---56# balsam-eval78> BALSAM: A Platform for Benchmarking Arabic Large Language Models — Al-Matham et al. (2025) (arXiv:2507.22603, 2025)910## What this evaluates1112Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning.1314## Datasets1516- **BALSAM** — total 52000; splits: test (-1), dev (-1); repo https://github.com/ksaa-nlp/balsam-eval1718## Metrics1920- `LLM as a judge` **(primary)** — range: [0, 3]21 - An LLM (Gemini 2.5 Flash) first extracts a concise answer from the model's verbose output, then rates it on a 0–3 scale mirroring human evaluation guidelines.2223## Input / output format2425**Input**: Arabic language prompts/tasks spanning 14 categories (e.g., Creative Writing, QA, Translation, Logic, Program Execution).2627**Output**: Model-generated text responses.2829## Scoring recipe3031```python32def score_llm_judge(predictions, gold):33 extracted = [llm_extract(p, prompt=EXTRACTION_PROMPT) for p in predictions]34 scores = [llm_rate(e, g, prompt=SCORING_PROMPT, scale=[0,1,2,3]) for e, g in zip(extracted, gold)]35 return sum(scores) / len(scores)36```3738## Common pitfalls3940- Traditional n-gram metrics (BLEU/ROUGE) heavily penalize verbose outputs and fail to handle Arabic's complex morphology, leading to artificially low scores for longer, correct answers.41- Model size is not a reliable predictor of Arabic capability; tokenization quality, Arabic training data volume, and Arabic-specific SFT are more critical factors.4243## Evidence (verbatim from paper)4445> Table [5] shows the correlation of human evaluation with ROUGE-LSum, BLEU, and BERTScore (with and without extraction of answers using an LLM) and LLM as a judge. We make the following observations: ... LLM as a judge was highly correlated with human judgments for all categories, with values ranging between 0.824 and 0.977.4647## Citation4849```bibtex50@misc{almatham2025balsam,51 title={BALSAM: A Platform for Benchmarking Arabic Large Language Models},52 author={Al-Matham et al. (2025)},53 year={2025},54 note={arXiv:2507.22603}55}56```5758- arXiv: 2507.22603
qhjqhj00