Mathwriting Eval

mathwriting-eval

MathWriting: A Dataset For Handwritten Mathematical Expression Recognition — Gervais et al. (2024) (arXiv:2404.10690, 2024)

What this evaluates

Evaluates a model's ability to recognize handwritten mathematical expressions and convert them into normalized LaTeX. It probes both offline (rasterized image) and online (ink coordinate) recognition capabilities, measuring how accurately the model reconstructs complex mathematical structures and symbols.

Datasets

• MathWriting — total 630000; splits: train (-1), valid (-1), test (-1)

Metrics

• CER (primary) — range: [0, 1]
  • Character Error Rate computed over LaTeX tokens rather than ASCII characters. Calculated as the minimum number of single-character edits (insertions, deletions, substitutions) required to transform the predicted LaTeX string into the ground truth LaTeX string, divided by the length of the ground truth string.
• Exact Match (EM) — range: [0, 1]
  • Binary metric indicating whether the predicted LaTeX string exactly matches the ground truth LaTeX string character-for-character.
• ≤ 1 dist — range: percent
  • Percentage of predictions where the Levenshtein edit distance between the predicted and ground truth LaTeX strings is less than or equal to 1.

Input / output format

Input: InkML format (sequence of coordinates with speed/time information) and/or rasterized bitmap images (black ink on white background).

Output: Normalized LaTeX string representing the mathematical expression.

Scoring recipe

def compute_cer(pred_latex, gold_latex):
    pred_tokens = tokenize_latex(pred_latex)  # Uses Appendix M tokenizer
    gold_tokens = tokenize_latex(gold_latex)
    dist = levenshtein_distance(pred_tokens, gold_tokens)
    return dist / len(gold_tokens)

Common pitfalls

• Using standard ASCII tokenization instead of the specified LaTeX tokenization, which unfairly penalizes Greek letters and mathematical symbols.
• Ignoring the distinction between online (ink coordinates) and offline (rasterized image) input formats, which drastically changes model performance.
• Not using the official reference implementation for tokenization and edit distance, leading to non-comparable CER scores across different runs.

Evidence (verbatim from paper)

We propose the following evaluation setup based on MathWriting for the quality of handwriting math expression recognition. - evaluation samples: the test split of MathWriting. - metric: character error rate (CER) [22], where a "character" is a LATEX token as defined by the code in Appendix M. We provide a reference implementation of the evaluation metric at the Github page 1 . We propose the use of CER as a metric to make results comparable to other recognition tasks like text recognition [26, 17], and the use of LATEX tokens instead of ASCII characters so that an error on a single non-latin letter (e.g. \alpha recognized as a) counts as one instead of many.

Citation

@misc{gervais2024mathwriting,
  title={MathWriting: A Dataset For Handwritten Mathematical Expression Recognition},
  author={Gervais et al. (2024)},
  year={2024},
  note={arXiv:2404.10690}
}

• arXiv: 2404.10690