Mwp Value Accuracy Eval

mwp-value-accuracy-eval

Math Word Problem Solving by Generating Linguistic Variants of Problem Statements — Raiyan et al. (2023) (arXiv:2306.13899, 2023)

What this evaluates

Evaluates mathematical reasoning and robustness on single-equation math word problems. It probes a model's ability to parse linguistic variations, ignore irrelevant information, and solve inverted or structurally complex problems.

Datasets

• MAWPS — total 2373; splits: 5-fold CV (2373)
• SVAMP — total 1000; splits: test (1000)
• PARAMAWPS — total 16278; splits: 5-fold CV (16278)

Metrics

• Value accuracy (primary) — range: percent
  • Percentage of problems where the model's predicted numerical answer exactly matches the ground truth value. Calculated as (number of correct predictions / total number of problems) × 100.

Input / output format

Input: Natural language math word problem statement.

Output: Numerical answer or mathematical expression. LLMs are instructed to output in the format 'Answer: [ANS]'.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_values):
    if str(pred).strip() == str(gold).strip():
        correct += 1
value_accuracy = (correct / len(predictions)) * 100

Common pitfalls

• Uses 5-fold cross-validation for fine-tuned models but zero-shot evaluation for LLMs; results are not directly comparable across settings.
• PARAMAWPS contains adversarial variations (e.g., unrelated information, inverted questions) that test robustness beyond standard template matching.
• The reported accuracy often includes a majority voting mechanism over k generated variants; evaluating single instances without voting will yield lower scores.

Evidence (verbatim from paper)

Table 3: Value accuracy of the DeBERTa model and various baseline models. † denotes 5-fold cross validation. ... The DeBERTa model coupled with the Paraphrasing model and the Voting Mechanism outperforms all the baseline models in the MAwPS (Koncel-Kedziorski et al., 2016) dataset with an accuracy of 91.0%.

Citation

@misc{raiyan2023mathwordproblem,
  title={Math Word Problem Solving by Generating Linguistic Variants of Problem Statements},
  author={Raiyan et al. (2023)},
  year={2023},
  note={arXiv:2306.13899}
}

• arXiv: 2306.13899