Fanar20 Benchmarks Eval

fanar20-benchmarks-eval

Fanar 2.0: Arabic Generative AI Stack — FANAR TEAM et al. (2026) (arXiv:2603.16397, 2026)

What this evaluates

Evaluates bilingual language understanding, reasoning, and instruction-following capabilities of generative AI models. It probes general knowledge, commonsense reasoning, and culturally aligned Arabic comprehension across multiple standard and custom benchmarks.

Datasets

• English Benchmarks (MMLU, HellaSwag, ARC-Challenge, PIQA, Winogrande) — total ?; splits: test (-1)
• OALL v1 — total ?; splits: test (-1)

Metrics

• English Avg. (primary) — range: percent
  • Arithmetic mean of accuracy scores across MMLU, HellaSwag, ARC-Challenge, PIQA, and Winogrande.
• Arabic Avg. (primary) — range: percent
  • Arithmetic mean of accuracy scores across the OALL v1 benchmark suite.

Input / output format

Input: Multiple-choice questions, factual QA prompts, and open-ended instruction-following prompts in English and Arabic.

Output: Model-generated text responses, including selected options for multiple-choice tasks or free-form answers/reasoning traces for open-ended tasks.

Scoring recipe

def compute_english_avg(predictions, golds):
    scores = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    return sum(scores) / len(scores) * 100

def compute_arabic_avg(predictions, golds):
    scores = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
    return sum(scores) / len(scores) * 100

# For open-ended generation (MT-Bench, Alpaca, 50+ domains):
# Score = automated_judge_model(prompt, response).score

Common pitfalls

• English and Arabic averages are computed over different benchmark suites, so direct cross-lingual comparison of absolute scores is invalid.
• Open-ended generation evaluation relies on automated closed-source LLM judges rather than human raters, which may introduce scoring bias.
• The exact composition and weighting of the 'composite benchmark score' used for in-loop monitoring are not fully disclosed.

Evidence (verbatim from paper)

English Avg. and Arabic Avg. are reported as the average across several English benchmarks, e.g., MMLU, HellaSwag, ARC-Challenge, PIQA, Winogrande; and the OALL v1 benchmark suite average, respectively. The suite covers instruction-following (translated MT-Bench [21], Alpaca benchmarks with custom judging prompts), broad general capabilities across 50+ domains, and multi-turn conversational fluency—all evaluated on open-ended generation using strong closed-source models as automated judges.

Citation

@misc{fanarteam2026fanar20,
  title={Fanar 2.0: Arabic Generative AI Stack},
  author={FANAR TEAM et al. (2026)},
  year={2026},
  note={arXiv:2603.16397}
}

• arXiv: 2603.16397