Llama Berry Eval

llama-berry-eval

LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning — Di Zhang et al. (2024) (arXiv:2410.02884, 2024)

What this evaluates

Evaluates LLMs on complex mathematical reasoning using search-based inference (SR-MCTS) rather than direct generation. It measures success rates across varying difficulty levels, from grade-school math to Olympiad-level problems, by testing both majority-vote and best-of-k strategies.

Datasets

• AIME24 — total 30; splits: test (30)
• AMC23 — total ?; splits: test (-1)
• Math Odyssey — total ?; splits: test (-1)
• GPQA Diamond — total ?; splits: test (-1)
• OlympiadBench — total ?; splits: test (-1)
• College Math — total ?; splits: test (-1)
• MMLU STEM — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• GSMHard — total ?; splits: test (-1)
• MATH500 — total ?; splits: test (-1)

Metrics

• major@k (primary) — range: percent
  • Majority voting accuracy over k inference rollouts. A problem is counted as solved if more than half of the k generated answers are graded correct.
• rm@k — range: percent
  • Resolve rate (best-of-k) over k inference rollouts. A problem is counted as solved if at least one of the k generated answers is graded correct.

Input / output format

Input: Textual mathematical problem statement.

Output: Answer string conforming to the format specified in the prompt. Graded as consistent if it exactly matches the ground truth, closely approximates it numerically, or is equivalent in symbolic form.

Scoring recipe

def evaluate(predictions, gold, k):
    correct = []
    for ans in predictions:
        if exact_match(ans, gold) or close_numerical(ans, gold) or symbolic_equivalent(ans, gold):
            correct.append(1)
        else:
            correct.append(0)
    major_k = 1 if sum(correct) > k / 2 else 0
    rm_k = 1 if sum(correct) > 0 else 0
    return major_k, rm_k

Common pitfalls

• Confusing major@k (majority vote) with rm@k (best-of-k/resolve rate), which yield different results especially on hard problems.
• Grading requires checking for numerical tolerance and symbolic equivalence, not just exact string matches.
• Performance is highly dependent on the number of inference-time rollouts (k), so results are not directly comparable across different k values.

Evidence (verbatim from paper)

We score answers as consistent if they exactly match the ground truth, closely approximate it numerically, or are equivalent in symbolic form. To ensure a comprehensive and rigorous evaluation, we adopt major@k (Kuncheva, [2014]) and rm@k (Yang et al., [2024c]), which can be unified as the solved rate of problems (Lightman et al., [2023]; Luo et al., [2024a]).

Citation

@misc{zhang2024llamaberry,
  title={LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning},
  author={Di Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.02884}
}

• arXiv: 2410.02884