Character Detection Matching

character-detection-matching

Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching — Bin Wang et al. (2024) (arXiv:2409.03643, 2024)

What this evaluates

Evaluates the visual fidelity and spatial accuracy of formula recognition models by comparing rendered images of predicted and ground-truth LaTeX code at the character level. It addresses the misalignment of text-based metrics with human perception by treating each character as a detectable object in an image.

Datasets

• UniMER-Test — total 23757; splits: test (23757)
• Tiny-Doc-Math — total 437; splits: test (437)

Metrics

• CDM (primary) — range: [0, 1]
  • Image-level character detection matching. Renders predicted and ground-truth LaTeX to images, detects characters as objects, and computes visual similarity with spatial localization. Rendering failures are assigned a score of 0.
• BLEU — range: [0, 1]
  • Standard n-gram based text similarity metric, normalized as per prior formula recognition literature.
• ExpRate — range: [0, 1]
  • Exact match rate based on character-level string comparison of predicted and ground-truth LaTeX.
• ExpRate@CDM — range: [0, 1]
  • Proportion of predictions that achieve a perfect CDM score, indicating completely correct formula recognition.

Input / output format

Input: Formula-level: cropped single formula images. Document-level: PDF pages or screenshots containing multiple formulas.

Output: Formula-level: predicted LaTeX string. Document-level: Markdown text containing recognized block formulas.

Scoring recipe

def compute_cdm(pred_latex, gt_latex):
    pred_img = render_latex(pred_latex)
    gt_img = render_latex(gt_latex)
    if pred_img is None:
        return 0.0
    pred_chars = detect_characters(pred_img)
    gt_chars = detect_characters(gt_img)
    matches = spatial_visual_match(pred_chars, gt_chars)
    return len(matches) / max(len(gt_chars), 1)

Common pitfalls

• Traditional text-based metrics (BLEU, ExpRate) are highly sensitive to LaTeX formatting and style variations, leading to misleading performance rankings across different models.
• Document-level evaluation relies on regex matching to extract block formulas from Markdown outputs, which can fail on non-standard or inline formatting.
• Rendering failures automatically yield a CDM score of 0, which may disproportionately penalize models that produce semantically correct but syntactically unrenderable LaTeX.

Evidence (verbatim from paper)

We validate the CDM metric by evaluating several mainstream formula recognition models using both subjective impressions and objective metrics. The models include open-source UniMERNet, Texify, Pix2tex, and the commercial Mathpix API, all tested on the UniMER-Test dataset. The CDM metric relies on the successful rendering of formula images. For models that fail to render images, we assign a CDM score of 0, as rendering failures indicate that the predicted LaTeX code lacks critical elements.

Citation

@misc{wang2024image,
  title={Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching},
  author={Bin Wang et al. (2024)},
  year={2024},
  note={arXiv:2409.03643}
}

• arXiv: 2409.03643