Romath Eval

romath-eval

RoMath: A Mathematical Reasoning Benchmark in Romanian — Cosma et al. (2024) (arXiv:2409.11074, 2024)

What this evaluates

This benchmark evaluates large language models' mathematical reasoning capabilities specifically in Romanian. It probes the ability to solve single-step and multi-step problems, handle verifiable numerical answers, and construct or verify mathematical proofs without relying on direct English translations.

Datasets

• RoMath — total 76910; splits: Baccalaureate_train (4300), Baccalaureate_test (1480), Competitions_train (804), Competitions_test (329), Synthetic_train (55900), Synthetic_test (7100); repo https://github.com/cosmaadrian/romath

Metrics

• correctness (primary) — range: [0, 1]
  • For verifiable problems, exact string match after normalization between the model's boxed final answer and the ground truth. For proof problems or missing boxes, an open-weight judge LLM scores the solution's correctness against the problem and gold solution.

Input / output format

Input: Romanian mathematical problem statement (provided in Markdown/LaTeX format), optionally including intermediate steps or proof requirements.

Output: Model-generated solution. For verifiable problems, the final answer must be enclosed in a \boxed{} tag.

Scoring recipe

def compute_correctness(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if gold.is_verifiable and '\\boxed{' in pred:
            pred_ans = extract_boxed(pred)
            if normalize(pred_ans) == normalize(gold.final_answer):
                correct += 1
        else:
            if judge_llm.evaluate(gold.problem, gold.solution, pred) >= 0.5:
                correct += 1
    return correct / len(predictions)

Common pitfalls

• Assuming direct translation of English math problems preserves semantic precision, which the paper shows causes significant performance drops.
• Expecting formal proof-checkers to handle natural-language proofs directly, whereas the benchmark relies on an LLM judge due to the open nature of this problem.
• Failing to format final answers in \boxed{} tags, which forces fallback to the less reproducible LLM judge for verifiable problems.

Evidence (verbatim from paper)

For evaluating verifiable problems, we adopt the procedure from Hendrycks et al. (2021) for string comparison after the solutions are normalized; this requires the model to output solutions in a \boxed{} tag. However, if the model does not provide the solution in this format or if the problem requires a proof, we employ a judge LLM to estimate correctness, inspired by several other works (Zheng et al. (2023); Fang et al. (2024)).

Citation

@misc{cosma2024romath,
  title={RoMath: A Mathematical Reasoning Benchmark in Romanian},
  author={Cosma et al. (2024)},
  year={2024},
  note={arXiv:2409.11074}
}

• arXiv: 2409.11074