Estonian Native LLM Benchmark Eval

estonian-native-llm-benchmark-eval

Estonian Native Large Language Model Benchmark — Lillepalu et al. (2025) (arXiv:2510.21193, 2025)

What this evaluates

Evaluates large language models on native Estonian language capabilities across seven tasks covering factual recall, grammar, morphology, vocabulary, summarization, and structured information extraction. The benchmark emphasizes cultural and linguistic authenticity by using human-curated or native-source data without machine translation, assessing both general and domain-specific competencies.

Datasets

• Exams — total ?; splits: test (-1)
• Trivia — total ?; splits: test (-1)
• Declension — total ?; splits: test (-1)
• Words — total ?; splits: test (-1)
• Grammar — total ?; splits: test (-1)
• News — total ?; splits: test (-1)
• Speaker Name Extraction — total ?; splits: test (-1)

Metrics

• Mean Score (primary) — range: [0, 1]
  • Arithmetic average of all normalized task scores (scaled 0–1) to enable cross-benchmark comparison.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted answers out of total instances for Exams, Trivia, Declension, and Words tasks.
• Levenshtein distance — range: [0, 1]
  • 1 / (1 + Levenshtein distance) between predicted and gold text for the Grammar task.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence recall/precision score between generated summary and reference for the News task.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for extracted speaker names in the Speaker Name Extraction task.

Input / output format

Input: Estonian-language prompts tailored to each task type, including multiple-choice questions (Exams/Trivia), word inflection/definition tasks (Declension/Words), grammar correction inputs, news transcripts or articles for summarization, and automatic transcripts with speaker codes for entity extraction.

Output: Varies by task: exact text or multiple-choice selection for knowledge/linguistic tasks, corrected text for grammar, free-form summary for news, and a strict JSON list of strings for speaker extraction.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task in ['Exams', 'Trivia', 'Declension', 'Words']:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task == 'Grammar':
        return sum(1 / (1 + levenshtein(p, g)) for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'News':
        return sum(rouge_l(p, g) for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'Speaker Name Extraction':
        scores = []
        for p, g in zip(predictions, golds):
            p_set, g_set = set(p), set(g)
            tp = len(p_set & g_set)
            prec = tp / len(p_set) if p_set else 0
            rec = tp / len(g_set) if g_set else 0
            f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
            scores.append(f1)
        return sum(scores) / len(scores)

Common pitfalls

• ROUGE-L may not reliably reflect actual model quality for news summarization due to semantic variability and noise in outputs.
• Speaker extraction prioritizes precision over recall; models should avoid guessing names to prevent false positives.
• Context length limits can cause certain models (e.g., Gemma variants) to fail completely on transcript-based tasks.

Evidence (verbatim from paper)

For all benchmarks, we used metrics scaled between 0 and 1, where higher values indicate better performance. This design allows for meaningful averaging of results across benchmarks. For the grammar correction benchmark, we employed a custom metric based on the Levenshtein distance, defined as 1 / (1 + Levenshtein distance), which also produces values in the range [0, 1] and is therefore compatible with the overall evaluation framework.

Citation

@misc{lillepalu2025estonian,
  title={Estonian Native Large Language Model Benchmark},
  author={Lillepalu et al. (2025)},
  year={2025},
  note={arXiv:2510.21193}
}

• arXiv: 2510.21193