Sahm Eval

sahm-eval

SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning — Elbadry et al. (2026) (arXiv:2604.19098, 2026)

What this evaluates

Evaluates Arabic language models on financial and Shari’ah-compliant reasoning across multiple task types, including multiple-choice questions, extractive summarization, and open-ended question answering. It probes the gap between general Arabic fluency and domain-specific procedural/financial reasoning.

Datasets

• SAHM — total 14380; splits: train (-1), test (-1); repo https://github.com/rania-hossam/SAHM

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • 1 if the normalized prediction matches the gold choice exactly, else 0. Free-form outputs (option text or letters) are normalized to a single canonical choice before comparison.
• ROUGE-F1 — range: [0, 1]
  • Standard ROUGE-F1 score computed over ROUGE-1, ROUGE-2, and ROUGE-L n-gram overlaps between the model's verbatim output and the gold extractive reference.
• LLM-as-a-judge score — range: [0, 10]
  • Additive [0,10] score generated by Gemini-2.5-Flash evaluating alignment with the reference ruling, preservation of constraints/quantitative fidelity, correctness, Arabic clarity, and directness/grounding.

Input / output format

Input: Arabic prompts with task-specific instructions (e.g., MCQ options, source text for summarization, or open-ended questions). For judge scoring, the original prompt, gold reference, and model answer are provided.

Output: For MCQs: normalized to a single choice (option letter or text). For summarization: verbatim sentences extracted from the source. For QA: free-form Arabic text.

Scoring recipe

def score_mcq(pred, gold):
    normalized_pred = normalize_to_single_choice(pred)
    return 1.0 if normalized_pred == gold else 0.0

def score_summarization(pred, gold):
    return rouge_f1(pred, gold) # ROUGE-1/2/L

def score_qa_judge(prompt, gold, pred):
    # Greedy decoding (temp=0) required
    score = judge_score(prompt, gold, pred, rubric="alignment, constraints, correctness, clarity, grounding")
    return score # [0, 10]

Common pitfalls

• Arabic fluency does not guarantee financial reasoning; models often score >90% on recognition tasks but drop to <10% on open-ended generation.
• MCQ outputs must be normalized to a single choice (letter or text) before exact-match scoring; raw free-form outputs will fail.
• LLM-as-a-judge requires greedy decoding (temperature 0) and fixed max lengths to ensure reproducibility across models.

Evidence (verbatim from paper)

We evaluate Accounting Exams, Business Exams, Fatwa MCQ, and Financial Sentiment with exact-match accuracy, normalizing free-form outputs (e.g., option text/letters) to a single choice before scoring (Appendix H). For extractive summarization, we report ROUGE-F1 (ROUGE-1/2/L) against gold extractive references (models are instructed to output verbatim sentences). For Fatwa QA, Shari’ah Standards QA, and Event-Cause QA, we use Gemini-2.5-Flash as an LLM-as-a-judge (blind to model identity): given the original Arabic prompt, gold reference, and model answer, it returns a JSON-validated, additive (sum-of-components) $[0,10]$ score under a shared rubric assessing alignment with the reference ruling/conclusion, preservation of key constraints or quantitative fidelity, correctness (doctrinal/factual or financial reasoning), Arabic clarity, and directness/grounding.

Citation

@misc{elbadry2026sahm,
  title={SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning},
  author={Elbadry et al. (2026)},
  year={2026},
  note={arXiv:2604.19098}
}

• arXiv: 2604.19098