Svamp Eval

svamp-eval

Are NLP Models really able to Solve Simple Math Word Problems? — Patel et al. (2021) (arXiv:2103.07191, 2021)

What this evaluates

Evaluates whether NLP models can genuinely solve simple math word problems through arithmetic reasoning versus relying on shallow heuristics like bag-of-words matching or positional cues. It probes model brittleness by testing performance on standard datasets alongside carefully perturbed variants that remove questions or alter operator types.

Datasets

• MAWPS — total ?; splits: (unstated)
• ASDiv-A — total ?; splits: (unstated)
• SVAMP — total ?; splits: (unstated); repo https://github.com/arkilpatel/SVAMP

Metrics

• accuracy (primary) — range: percent
  • Exact-match correctness averaged over 5-fold cross-validation splits.

Input / output format

Input: Natural language math word problem statements (sometimes modified to remove the question sentence or alter word order).

Output: A single numerical value representing the solution to the word problem.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if float(pred) == float(gold):
        correct += 1
accuracy = (correct / len(gold_answers)) * 100

Common pitfalls

• Models often achieve high accuracy on standard benchmarks by exploiting shallow heuristics (e.g., bag-of-words features or word order) rather than performing actual arithmetic.
• Performance drops significantly when the question sentence is removed or when problems are rephrased, indicating a lack of robust compositional reasoning.

Evidence (verbatim from paper)

The 5-fold cross-validation accuracies of the Transformer on MAWPS and ASDiv-A are provided in Table 16. The scores on Question-removed datasets are provided in Table 17 and on SVAMP challenge set is provided in Table 18.

Citation

@misc{patel2021svamp,
  title={Are NLP Models really able to Solve Simple Math Word Problems?},
  author={Patel et al. (2021)},
  year={2021},
  note={arXiv:2103.07191}
}

• arXiv: 2103.07191