qhjqhj00 hrm8k-eval
Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap — Ko et al. (2025) (arXiv:2501.02448, 2025)
What this evaluates
Evaluates multilingual mathematical reasoning capability, specifically probing whether models can comprehend and solve Korean math problems by leveraging English-as-pivot reasoning to bridge cross-lingual comprehension gaps.
Datasets
- HRM8K — total ?; splits: test (-1)
Metrics
pass@1(primary) — range: percent- Percentage of problems where the model's single generated response matches the ground-truth answer. Evaluated in a pass@1 setting.
Input / output format
Input: Korean-language mathematical word problems or equations.
Output: Step-by-step reasoning and final answer, constrained to be generated in Korean.
Scoring recipe
def compute_pass1(predictions, gold):
correct = 0
for pred, g in zip(predictions, gold):
if extract_final_answer(pred) == g:
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Using extremely low sampling temperatures causes models to revert to English or Chinese instead of the target language.
- Multilingual performance gaps are frequently misattributed to reasoning limitations rather than input comprehension difficulties.
Evidence (verbatim from paper)
While lower temperatures are often used in pass@1 settings, we observed that extremely low temperatures sometimes cause models to revert to their preferred language (often English or Chinese). Hence, to maintain the specified response language, we employ a slightly higher temperature with moderate top_p.
Citation
@misc{ko2025understand,
title={Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap},
author={Ko et al. (2025)},
year={2025},
note={arXiv:2501.02448}
}
- arXiv: 2501.02448