# Gamayun Eval

> Evaluates multilingual LLM capabilities across general knowledge, reasoning, mathematics, and cultural understanding in English, Russian, and other languages. It probes zero-shot and few-shot performance on standardized benchmarks and custom cultural knowledge tests. Use when the user wants to benchmark on MMLU, GSM8K, MERA, RuBIN, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/gamayun-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/gamayun-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/gamayun-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/gamayun-eval

---


# gamayun-eval

> Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM — Podolskiy et al. (2025) (arXiv:2512.21580, 2025)

## What this evaluates

Evaluates multilingual LLM capabilities across general knowledge, reasoning, mathematics, and cultural understanding in English, Russian, and other languages. It probes zero-shot and few-shot performance on standardized benchmarks and custom cultural knowledge tests.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **MERA** — total ?; splits: test (-1)
- **RuBIN** — total 5000; splits: test (5000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice or closed-form questions. Calculated as (correct predictions / total instances) * 100.
- `MERA score` — range: percent
  - Average performance across 15 tasks assessing different LLM abilities, using closed test subsets.
- `RuBIN score` — range: percent
  - Accuracy on a 5,000-item multiple-choice benchmark covering culturally specific Russian topics (proverbs, riddles, songs, films).

## Input / output format

**Input**: Multiple-choice questions or open-ended prompts, typically provided with few-shot examples (e.g., 5-shot) or zero-shot instructions.

**Output**: Model generates text completion or selects the correct option from provided choices.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if normalize(p) == normalize(g))
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using different shot counts than specified (e.g., 5-shot vs 0-shot) can drastically change scores.
- MERA and RuBIN are not publicly available or are internal/closed benchmarks, making direct replication difficult.
- Aggregating scores across languages/tasks without weighting can mask performance distributional skew.

## Evidence (verbatim from paper)

> We use zero-shot evaluation in all cases unless explicitly stated otherwise. ... The MERA score is calculated as the average performance across 15 tasks assessing different LLM abilities, using closed test subsets.

## Citation

```bibtex
@misc{podolskiy2025gamayun,
  title={Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM},
  author={Podolskiy et al. (2025)},
  year={2025},
  note={arXiv:2512.21580}
}
```

- arXiv: 2512.21580