qhjqhj00 crohme-hme100k-eval
Syntax-Aware Network for Handwritten Mathematical Expression Recognition — Ye Yuan et al. (2022) (arXiv:2203.01601, 2022)
What this evaluates
Evaluates handwritten mathematical expression recognition (HMER) by measuring exact LaTeX sequence matching, tolerant symbol-level error rates, and structural tree prediction accuracy on complex handwritten formulas.
Datasets
- CROHME — total 8836; splits: train (8836), test_2014 (986), test_2016 (1147), test_2019 (1199)
- HME100K — total 24607; splits: test_easy (7721), test_moderate (10450), test_hard (6436); repo https://github.com/tal-tech/SAN
Metrics
ExpRate(primary) — range: percent- Percentage of predicted mathematical expressions that exactly match the ground truth LaTeX sequence.
≤1— range: percent- Percentage of predictions tolerating at most one symbol-level error compared to ground truth.
≤2— range: percent- Percentage of predictions tolerating at most two symbol-level errors compared to ground truth.
ESPR— range: percent- Percentage of expressions whose structural tree is recognized correctly, irrespective of symbol labels.
Input / output format
Input: Handwritten mathematical expression images converted from InkML stroke trajectory data.
Output: Predicted LaTeX markup sequence representing the mathematical expression.
Scoring recipe
def compute_metrics(predictions, golds):
exact = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
err1 = sum(1 for p, g in zip(predictions, golds) if symbol_edit_distance(p, g) <= 1) / len(golds)
err2 = sum(1 for p, g in zip(predictions, golds) if symbol_edit_distance(p, g) <= 2) / len(golds)
struct = sum(1 for p, g in zip(predictions, golds) if parse_structure(p) == parse_structure(g)) / len(golds)
return exact * 100, err1 * 100, err2 * 100, struct * 100
Common pitfalls
- ExpRate requires exact string match, while ≤1/≤2 allow symbol-level deviations; confusing them leads to incorrect benchmark reporting.
- ESPR evaluates only the tree structure (syntax), ignoring symbol labels, which differs from standard sequence matching metrics.
- HME100K test splits are difficulty-stratified by Structural Complexity and Character Length, not randomly sampled.
Evidence (verbatim from paper)
Recognition Protocol. Expression recognition rate (ExpRate) is the widely used recognition protocol for mathematical expression recognition, defined as the percentage of predicted mathematical expressions accurately matching the ground truth. ExpRate ≤ 1 and ≤ 2 indicate the expression recognition rates are tolerable at most one or two symbol-level errors. Structure Recognition Protocol. Expression Structure Prediction Rate (ESPR) is used as the structure recognition protocol. ESPR is calculated by the percent of MEs whose structure is recognized correctly irrespective of symbol labels.
Citation
@misc{yuan2022syntax,
title={Syntax-Aware Network for Handwritten Mathematical Expression Recognition},
author={Ye Yuan et al. (2022)},
year={2022},
note={arXiv:2203.01601}
}
- arXiv: 2203.01601