# Mera Eval

> Evaluates large language models on Russian-language instruction following across 21 tasks spanning 11 skill domains, including problem-solving, exam-based questions, and ethical diagnostics. It probes zero-shot and few-shot capabilities under strict black-box conditions to measure alignment with human performance and prevent data leakage. Use when the user wants to benchmark on MERA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/mera-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mera-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mera-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mera-eval

---


# mera-eval

> MERA: A Comprehensive LLM Evaluation in Russian — Fenogenova et al. (2024) (arXiv:2401.04531, 2024)

## What this evaluates

Evaluates large language models on Russian-language instruction following across 21 tasks spanning 11 skill domains, including problem-solving, exam-based questions, and ethical diagnostics. It probes zero-shot and few-shot capabilities under strict black-box conditions to measure alignment with human performance and prevent data leakage.

## Datasets

- **MERA** — total ?; splits: test (-1); repo https://github.com/ai-forever/MERA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances out of the total number of instances in a task. Calculated as (number of correct predictions / total predictions) * 100.

## Input / output format

**Input**: Russian-language instruction prompts or exam questions provided in zero-shot or few-shot format.

**Output**: Model-generated text response to the prompt.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
    total += 1
return (correct / total) * 100 if total > 0 else 0
```

## Common pitfalls

- Evaluations are conducted under fixed black-box conditions to strictly prevent data leakage, so fine-tuning on benchmark data invalidates results.
- Most models perform near-random on complex logic and reasoning tasks, indicating the benchmark is highly challenging for current architectures.
- Ethical diagnostic tasks (e.g., ruEthics) show no meaningful correlation with other capabilities, requiring separate safety-focused evaluation.

## Evidence (verbatim from paper)

> Moreover, they show prominent abilities on the arithmetic task SimpleAr exceeding 90% accuracy with the best score of 95.1 achieved by Yi-6B.

## Citation

```bibtex
@misc{fenogenova2024mera,
  title={MERA: A Comprehensive LLM Evaluation in Russian},
  author={Fenogenova et al. (2024)},
  year={2024},
  note={arXiv:2401.04531}
}
```

- arXiv: 2401.04531