# Balsam Eval

> Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning. Use when the user wants to benchmark on BALSAM, or asks about evaluating this task. Reports LLM as a judge.

- Skill: `qhjqhj00/balsam-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/balsam-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/balsam-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/balsam-eval

---


# balsam-eval

> BALSAM: A Platform for Benchmarking Arabic Large Language Models — Al-Matham et al. (2025) (arXiv:2507.22603, 2025)

## What this evaluates

Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning.

## Datasets

- **BALSAM** — total 52000; splits: test (-1), dev (-1); repo https://github.com/ksaa-nlp/balsam-eval

## Metrics

- `LLM as a judge` **(primary)** — range: [0, 3]
  - An LLM (Gemini 2.5 Flash) first extracts a concise answer from the model's verbose output, then rates it on a 0–3 scale mirroring human evaluation guidelines.

## Input / output format

**Input**: Arabic language prompts/tasks spanning 14 categories (e.g., Creative Writing, QA, Translation, Logic, Program Execution).

**Output**: Model-generated text responses.

## Scoring recipe

```python
def score_llm_judge(predictions, gold):
    extracted = [llm_extract(p, prompt=EXTRACTION_PROMPT) for p in predictions]
    scores = [llm_rate(e, g, prompt=SCORING_PROMPT, scale=[0,1,2,3]) for e, g in zip(extracted, gold)]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Traditional n-gram metrics (BLEU/ROUGE) heavily penalize verbose outputs and fail to handle Arabic's complex morphology, leading to artificially low scores for longer, correct answers.
- Model size is not a reliable predictor of Arabic capability; tokenization quality, Arabic training data volume, and Arabic-specific SFT are more critical factors.

## Evidence (verbatim from paper)

> Table [5] shows the correlation of human evaluation with ROUGE-LSum, BLEU, and BERTScore (with and without extraction of answers using an LLM) and LLM as a judge. We make the following observations: ... LLM as a judge was highly correlated with human judgments for all categories, with values ranging between 0.824 and 0.977.

## Citation

```bibtex
@misc{almatham2025balsam,
  title={BALSAM: A Platform for Benchmarking Arabic Large Language Models},
  author={Al-Matham et al. (2025)},
  year={2025},
  note={arXiv:2507.22603}
}
```

- arXiv: 2507.22603