Bigbench Arithmetic Eval

bigbench-arithmetic-eval

Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks — Liu et al. (2023) (arXiv:2305.14201, 2023)

What this evaluates

Evaluates large language models on arithmetic reasoning across addition, subtraction, multiplication, and division tasks with varying digit lengths. It probes the model's ability to handle large-number computation, number tokenization consistency, and stepwise reasoning without relying on external tools.

Datasets

• BIG-bench arithmetic — total ?; splits: test (-1)
• Extra arithmetic tasks — total ?; splits: test (-1)

Metrics

• exact string match (primary) — range: percent
  • 1 if the model's generated output exactly matches the ground truth string, else 0. Reported as a percentage across all instances.
• digit match — range: percent
  • Measures the proportion of correctly aligned digits between the predicted and ground truth answers, reflecting the per-token error rate. Reported as a percentage.

Input / output format

Input: Arithmetic problem statements (e.g., '12345 + 67890 = ?') with specified digit lengths (1D to 16D). For chain-of-thought evaluation, 'Solve it step by step' is appended to the prompt.

Output: Numerical answer string. For division tasks with remainders (extra tasks), the format is 'quotient R remainder'.

Scoring recipe

def exact_match(pred, gold):
    return 1.0 if pred.strip() == gold.strip() else 0.0

def digit_match(pred, gold):
    pred_digits = [c for c in pred if c.isdigit()]
    gold_digits = [c for c in gold if c.isdigit()]
    matches = sum(1 for p, g in zip(pred_digits, gold_digits) if p == g)
    total = max(len(pred_digits), len(gold_digits))
    return matches / total if total > 0 else 0.0

Common pitfalls

• Exact string match yields near-zero scores for models with inconsistent number tokenization, even when most digits are correct.
• Chain-of-thought prompting does not reliably improve accuracy because intermediate steps may contain errors that cancel out or lead to wrong final answers.
• BIG-bench division tasks only include cases without remainders, while custom tasks require 'quotient R remainder' formatting, causing evaluation mismatches if prompts are not adapted.

Evidence (verbatim from paper)

We first compute the accuracy based on the standard exact string match (Appendix C). We observe that GPT-4's accuracy under exact string match is almost identically zero on tasks involving large numbers. However, in many cases where the final answer is incorrect, the majority of digits in the generated answer align with the target number, with only a few digits being incorrect. Inspired by recent study on the emergent abilities of LLMs (Schaeffer et al., 2023), we include a digit match metric that can reflect the per-token error rate of the output, as each digit is uniquely represented by a token in LLaMA.

Citation

@misc{liu2023goat,
  title={Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2305.14201}
}

• arXiv: 2305.14201