Minerva Quant Reasoning Eval

minerva-quant-reasoning-eval

Solving Quantitative Reasoning Problems with Language Models — Lewkowycz et al. (2022) (arXiv:2206.14858, 2022)

What this evaluates

Evaluates large language models on quantitative reasoning and mathematical problem-solving tasks, including arithmetic, multi-step word problems, and standardized test questions, using few-shot prompting and chain-of-thought reasoning.

Datasets

• MATH — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• GSM8k — total ?; splits: test (-1)
• National Math Exam in Poland — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly solved problems out of the total number of problems in the dataset.
• MMLU highest score — range: percent
  • The score of the answer choice with the highest probability or score among the options, evaluated using a 5-shot prompt per topic.
• majority voting accuracy — range: percent
  • Accuracy computed by sampling k=16 model answers using a chain-of-thought prompt and taking the most frequent answer as the final prediction.

Input / output format

Input: Few-shot prompts (e.g., 5-shot per topic for MMLU) with chain-of-thought instructions.

Output: Chain-of-thought reasoning steps followed by a final answer or LaTeX-formatted equation.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def majority_voting_accuracy(predictions_list, gold):
    final_preds = [Counter(p).most_common(1)[0][0] for p in predictions_list]
    return compute_accuracy(final_preds, gold)

Common pitfalls

• MMLU evaluation requires topic-specific 5-shot prompts rather than a single global prompt.
• Majority voting requires sampling k=16 answers with chain-of-thought prompts instead of using greedy decoding.

Evidence (verbatim from paper)

The model achieves over $80%$ accuracy on 10-digit addition and over $20%$ accuracy on 18-digit addition.

Citation

@misc{lewkowycz2022solving,
  title={Solving Quantitative Reasoning Problems with Language Models},
  author={Lewkowycz et al. (2022)},
  year={2022},
  note={arXiv:2206.14858}
}

• arXiv: 2206.14858