Gaeleval Eval

gaeleval-eval

GaelEval: Benchmarking LLM Performance for Scottish Gaelic — Devine et al. (2026) (arXiv:2604.02135, 2026)

What this evaluates

Evaluates LLMs' morphosyntactic competence, machine translation quality, and culturally grounded question-answering abilities in Scottish Gaelic. It probes how well models handle minority language grammar, idiomatic usage, and domain-specific cultural knowledge without relying on English-centric prompting.

Datasets

• GaelEval — total 2115; splits: test (2115); repo https://github.com/Peter-Devine/gaeleval

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of outputs that conform to the required JSON schema and contain the exact correct answer option.
• BLEU — range: [0, 1]
  • Standard n-gram precision score for machine translation evaluation, computed using SacreBLEU.
• chrF — range: [0, 1]
  • Character n-gram F-score measuring overlap between reference and hypothesis at the character level.

Input / output format

Input: Task-specific prompts: (1) MCQA: Gaelic sentence with a single blank + 4 labeled options; (2) Translation: English source text; (3) Q&A: Gaelic cultural context + question + 4 labeled options. All include a fixed system instruction and few-shot examples to enforce output format.

Output: Single-letter option (e.g., 'b. an fhuil') or JSON schema with a single key-value pair containing the answer. No explanations or additional punctuation.

Scoring recipe

# Accuracy (MCQA & Q&A)
correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
accuracy = correct / len(golds)

# Translation
bleu = sacrebleu.corpus_bleu(predictions, [golds])
chrf = sacrebleu.corpus_chrf(predictions, [golds])

Common pitfalls

• Dialectal variation in Scottish Gaelic morphosyntax means some MCQA items may be ambiguous or admit multiple acceptable responses across regions.
• Human baseline participants were recruited via convenience sampling (N=35) and combine near-native/advanced speakers, which may overestimate population-level fluency.
• API rate limits and transient failures require exponential back-off and retry logic; ignoring this can lead to biased missing data or incorrect accuracy scores.

Evidence (verbatim from paper)

For the MCQA tasks, we report accuracy, defined as the percentage of outputs that both conformed to the required JSON schema (see Section 3.2) and contained the correct answer. For translation, we report BLEU (Papineni et al., 2002) and chrF (Popović, 2015).

Citation

@misc{devine2026gaeleval,
  title={GaelEval: Benchmarking LLM Performance for Scottish Gaelic},
  author={Devine et al. (2026)},
  year={2026},
  note={arXiv:2604.02135}
}

• arXiv: 2604.02135