Greekmmlu Eval

greekmmlu-eval

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek — Zhang et al. (2026) (arXiv:2602.05150, 2026)

What this evaluates

This benchmark evaluates large language models' ability to answer multiple-choice questions across 45 diverse academic, professional, and governmental subjects in Greek. It specifically probes native language fluency, cultural grounding, and domain-specific knowledge retention under zero-shot and few-shot prompting conditions.

Datasets

• GreekMMLU — total 21805; splits: test (-1); repo https://github.com/mersinkonomi/GreekMMLU

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For open-weight models, the option with the highest token log-likelihood is selected. For closed-source APIs, the model generates free-form text and the predicted Greek label is extracted via regular expressions.

Input / output format

Input: Greek multiple-choice question with subject-specific instruction, question stem, and four labeled options (A, B, Γ, Δ). Prompts are entirely in Greek and may include five representative in-context examples for the five-shot setting.

Output: For open-weight models: implicit selection via log-likelihood ranking. For closed-source models: direct extraction of the Greek answer key (A, B, Γ, Δ) from generated text using regular expressions.

Scoring recipe

correct = 0
for instance in dataset:
    if model_type == 'open_weight':
        pred = argmax([log_likelihood(instance, opt) for opt in instance.options])
    else:
        pred = extract_regex(model.generate(instance.prompt))
    if pred == instance.gold_label:
        correct += 1
return (correct / len(dataset)) * 100

Common pitfalls

• Confusing Greek answer labels (A, B, Γ, Δ) with standard Latin letters (A, B, C, D) during regex extraction.
• Assuming five-shot prompting universally improves performance; it only helps models >2B parameters, while smaller models remain near random baseline.
• Overlooking the distinction between public and private subsets when comparing results or citing performance.

Evidence (verbatim from paper)

Gemini 3 Flash reaches an average accuracy of 93.16%, while GPT-5.2 and GPT-4o achieve 87.75% and 86.81%, respectively, consistently excelling in all subjects.

Citation

@misc{zhang2026greekmmlu,
  title={GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.05150}
}

• arXiv: 2602.05150