Gamayun Eval

gamayun-eval

Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM — Podolskiy et al. (2025) (arXiv:2512.21580, 2025)

What this evaluates

Evaluates multilingual LLM capabilities across general knowledge, reasoning, mathematics, and cultural understanding in English, Russian, and other languages. It probes zero-shot and few-shot performance on standardized benchmarks and custom cultural knowledge tests.

Datasets

• MMLU — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MERA — total ?; splits: test (-1)
• RuBIN — total 5000; splits: test (5000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice or closed-form questions. Calculated as (correct predictions / total instances) * 100.
• MERA score — range: percent
  • Average performance across 15 tasks assessing different LLM abilities, using closed test subsets.
• RuBIN score — range: percent
  • Accuracy on a 5,000-item multiple-choice benchmark covering culturally specific Russian topics (proverbs, riddles, songs, films).

Input / output format

Input: Multiple-choice questions or open-ended prompts, typically provided with few-shot examples (e.g., 5-shot) or zero-shot instructions.

Output: Model generates text completion or selects the correct option from provided choices.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Using different shot counts than specified (e.g., 5-shot vs 0-shot) can drastically change scores.
• MERA and RuBIN are not publicly available or are internal/closed benchmarks, making direct replication difficult.
• Aggregating scores across languages/tasks without weighting can mask performance distributional skew.

Evidence (verbatim from paper)

We use zero-shot evaluation in all cases unless explicitly stated otherwise. ... The MERA score is calculated as the average performance across 15 tasks assessing different LLM abilities, using closed test subsets.

Citation

@misc{podolskiy2025gamayun,
  title={Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM},
  author={Podolskiy et al. (2025)},
  year={2025},
  note={arXiv:2512.21580}
}

• arXiv: 2512.21580