Symbolic Math Eval

symbolic-math-eval

Deep Learning for Symbolic Mathematics — Lample et al. (2019) (arXiv:1912.01412, 2019)

What this evaluates

This benchmark evaluates a model's ability to perform symbolic mathematical computations, specifically indefinite integration and solving ordinary differential equations. It probes the model's capacity to learn complex algebraic patterns and generate syntactically valid, mathematically equivalent expressions from prefix-encoded inputs.

Datasets

• Symbolic Mathematics (FWD/BWD/IBP/ODE) — total ?; splits: train (-1), test (5000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of test equations for which at least one hypothesis in the beam search is mathematically equivalent to the reference solution. For integration, equivalence is verified by differentiating the hypothesis and comparing it to the input function. For ODEs, equivalence is verified by substituting the hypothesis into the equation and checking if it evaluates to zero.

Input / output format

Input: Prefix-encoded mathematical expressions representing functions to integrate or ordinary differential equations.

Output: Prefix-encoded solution expressions (antiderivatives or general solutions to differential equations).

Scoring recipe

correct = 0
for preds, gold in zip(predictions, golds):
    solved = False
    for hyp in preds:
        if not is_valid_prefix(hyp):
            continue
        if is_symbolically_equivalent(hyp, gold):
            solved = True
            break
    if solved:
        correct += 1
return (correct / len(predictions)) * 100

Common pitfalls

• Relying on exact string matching instead of symbolic equivalence; the model often outputs algebraically rearranged forms or different constant-of-integration values that are mathematically correct.
• Using greedy decoding (beam size 1) for ODE tasks; accuracy drops significantly compared to beam search, as wider beams are critical for exploring the solution space.
• Ignoring invalid prefix expressions; the model may generate syntactically invalid sequences, which must be explicitly filtered out and counted as incorrect.

Evidence (verbatim from paper)

For the three problems, we measure the accuracy of our model on equations from the test set. Since we can easily verify the correctness of generated expressions, we consider all hypotheses in the beam, and not only the one with the highest score. We verify the correctness of each hypothesis, and consider that the model successfully solved the input equation if one of them is correct.

Citation

@misc{lample2019deeplearning,
  title={Deep Learning for Symbolic Mathematics},
  author={Lample et al. (2019)},
  year={2019},
  note={arXiv:1912.01412}
}

• arXiv: 1912.01412