# Hrm8k Eval

> Evaluates multilingual mathematical reasoning capability, specifically probing whether models can comprehend and solve Korean math problems by leveraging English-as-pivot reasoning to bridge cross-lingual comprehension gaps. Use when the user wants to benchmark on HRM8K, or asks about evaluating this task. Reports pass@1.

- Skill: `qhjqhj00/hrm8k-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/hrm8k-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/hrm8k-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/hrm8k-eval

---


# hrm8k-eval

> Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap — Ko et al. (2025) (arXiv:2501.02448, 2025)

## What this evaluates

Evaluates multilingual mathematical reasoning capability, specifically probing whether models can comprehend and solve Korean math problems by leveraging English-as-pivot reasoning to bridge cross-lingual comprehension gaps.

## Datasets

- **HRM8K** — total ?; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: percent
  - Percentage of problems where the model's single generated response matches the ground-truth answer. Evaluated in a pass@1 setting.

## Input / output format

**Input**: Korean-language mathematical word problems or equations.

**Output**: Step-by-step reasoning and final answer, constrained to be generated in Korean.

## Scoring recipe

```python
def compute_pass1(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        if extract_final_answer(pred) == g:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Using extremely low sampling temperatures causes models to revert to English or Chinese instead of the target language.
- Multilingual performance gaps are frequently misattributed to reasoning limitations rather than input comprehension difficulties.

## Evidence (verbatim from paper)

> While lower temperatures are often used in pass@1 settings, we observed that extremely low temperatures sometimes cause models to revert to their preferred language (often English or Chinese). Hence, to maintain the specified response language, we employ a slightly higher temperature with moderate top_p.

## Citation

```bibtex
@misc{ko2025understand,
  title={Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap},
  author={Ko et al. (2025)},
  year={2025},
  note={arXiv:2501.02448}
}
```

- arXiv: 2501.02448