Dialectalarabicmmlu Eval

dialectalarabicmmlu-eval

DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models — Altakrori et al. (2025) (arXiv:2510.27543, 2025)

What this evaluates

Evaluates large language models' ability to understand and reason across multiple Arabic dialects and standard Arabic across diverse academic and professional domains. It measures dialectal generalization and sensitivity to linguistic context by comparing performance under default, dialect-conditioned, and dialect-identification prompts.

Datasets

• DialectalArabicMMLU — total 21945; splits: test (21945)

Metrics

• accuracy (primary) — range: [0, 1]
  • Mean proportion of correctly predicted instances. The model selects the multiple-choice option with the highest log-likelihood; this prediction is correct if it matches the gold answer. Values are averaged over 32 domains and 5 inference runs.

Input / output format

Input: Multiple-choice question prompt. Format varies by setting: Default ('The following are multiple choice questions (with answers) about [domain]. [Question] [Choices] Answer:'), Oracle (adds dialect specification, e.g., '...in an Egyptian dialect.'), or Dialect Identification (fixed prompt for dialect classification).

Output: The model outputs the text of the selected multiple-choice option corresponding to the highest log-likelihood.

Scoring recipe

correct = 0
total = 0
for run in range(5):
    for item in dataset:
        log_probs = []
        for option in item.choices:
            prompt_with_option = item.prompt + ' ' + option
            log_probs.append(model.loglikelihood(prompt_with_option))
        predicted = item.choices[argmax(log_probs)]
        if predicted == item.gold:
            correct += 1
        total += 1
accuracy = correct / total

Common pitfalls

• Using sampling/generation instead of log-likelihood evaluation will yield different results due to temperature/top-p settings.
• Failing to average over 5 runs and 32 domains obscures variance and domain-specific biases.
• Ignoring the Oracle vs. Default setting difference masks the model's true dialectal generalization capability.
• Random chance baseline is 25% (4 options), not 0% or 100%.

Evidence (verbatim from paper)

The model inference is performed using the log-likelihood evaluation mode, where each answer option is appended to the input prompt and the log-likelihood of that option appearing after the given prompt is output by the model. The option with the highest log-likelihood is chosen as the model’s answer, and this prediction is deemed correct if it matches the correct answer for that question. Finally, we repeat each experiment five times, and we report the average accuracy, computed as the mean proportion of correctly predicted instances across all test items in all five runs.

Citation

@misc{altakrori2025dialectalarabicmmlu,
  title={DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models},
  author={Altakrori et al. (2025)},
  year={2025},
  note={arXiv:2510.27543}
}

• arXiv: 2510.27543