Numericbench Eval

numericbench-eval

Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models — Haoyang Li et al. (arXiv:2502.11075, 2025)

What this evaluates

This benchmark probes fundamental numerical abilities in large language models, including number recognition, arithmetic operations, contextual retrieval, comparison, summarization, and logical reasoning. It evaluates how well models handle structured and unstructured numerical data across varying context lengths and noise levels.

Datasets

• NumericBench — total ?; splits: test (-1); repo https://github.com/TreeAI-Lab/NumericBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For multiple-choice tasks, exact match to the single choice (A, B, C, etc.). For arithmetic tasks, computed answer rounded to two decimal places. For mixed-number-string tasks, exact string match.

Input / output format

Input: Text prompts containing numerical contexts (e.g., number lists, stock/weather tables, arithmetic expressions, or mixed alphanumeric strings) followed by a question requiring retrieval, comparison, summary, logic, or computation.

Output: Single choice letter (e.g., A, B, C) for classification/retrieval tasks, or exact numerical/string answer for arithmetic and recognition tasks.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = pred.strip().lower()
        gold_clean = gold.strip().lower()
        if pred_clean == gold_clean:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Models often fail on long contexts or noisy attributes, causing output disruption (scores marked with * in tables reflect short-subset evaluation).
• Tokenizers split numbers into multiple tokens, leading to poor number recognition and retrieval accuracy.
• Arithmetic evaluation requires answers rounded to exactly two decimal places; failing to apply this tolerance causes false negatives.
• The random baseline is 12.5% (8 options), not 0%, so low scores may still be above chance.

Evidence (verbatim from paper)

The evaluation metric is accuracy. Also, we set the exact answer for mixed-number-string dataset, set the computed answer to two decimal places for arithmetic datasets, and set the answer of each question as a single choice (e.g., A, B, or C) for other datasets to reliably evaluate LLMs*(Bai et al., [2024])*.

Citation

@misc{li2025exposing,
  title={Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models},
  author={Haoyang Li et al.},
  year={2025},
  note={arXiv:2502.11075}
}

• arXiv: 2502.11075