Formula Extraction Eval

formula-extraction-eval

Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs — Horn et al. (2025) (arXiv:2512.09874, 2025)

What this evaluates

Evaluates the ability of PDF document parsers to accurately extract mathematical formulas and preserve their semantic meaning. It probes format variability handling, representational non-uniqueness, and semantic equivalence recognition beyond simple character matching.

Datasets

• PDF Formula Extraction Benchmark — total 250; splits: test (250); repo https://github.com/phorn1/pdf-parse-bench

Metrics

• LLM-as-a-Judge (primary) — range: 0-10
  • Scores formula pairs on a 0–10 scale based on correctness, completeness, and semantic equivalence. Evaluated using GPT-5-mini.
• CDM — range: 0-1
  • Computes character-level precision, recall, and F1-score by rendering formulas, detecting bounding boxes, and performing bipartite matching based on token identity, spatial proximity, and sequential order.

Input / output format

Input: Ground truth mathematical formula and parsed formula output from a PDF parser.

Output: A numerical score from 0 to 10.

Scoring recipe

def score_formula_pair(gt_formula, parsed_formula, model='gpt-5-mini'):
    prompt = f'Score 0-10 on correctness, completeness, semantic equivalence:\nGT: {gt_formula}\nParsed: {parsed_formula}'
    response = model.generate(prompt)
    score = extract_score(response)  # Parse 0-10 float from LLM output
    return score

Common pitfalls

• Assuming text-based metrics (BLEU, Levenshtein) are sufficient; they fail due to format variability and representational non-uniqueness in LaTeX/MathML.
• Relying solely on CDM, which produces false positives for structural errors (superscripts/subscripts) and false negatives for Unicode outputs or semantically equivalent LaTeX variants.

Evidence (verbatim from paper)

We evaluated five contemporary LLM models (GPT-5, GPT-5-mini, GPT-5-nano, Gemini-2.5-Flash, and Mistral-Medium-2508), prompting them to score formula pairs on a 0–10 scale based on correctness, completeness, and semantic equivalence. As shown in Figure[2], all models exhibited substantially higher Pearson correlations with human scores (0.69–0.78) than CDM or text-based metrics, with GPT-5 and GPT-5-mini achieving the strongest correlations (0.78). We selected GPT-5-mini as our evaluation model given its competitive performance and cost-effectiveness. Although LLM evaluation is not infallible—occasionally assigning imperfect scores to identically rendered formulas—its correlation with human judgment (0.78) substantially exceeds that of CDM (0.34), making it a more suitable metric for evaluating mathematical formula extraction.

Citation

@misc{horn2025benchmarking,
  title={Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs},
  author={Horn et al. (2025)},
  year={2025},
  note={arXiv:2512.09874}
}

• arXiv: 2512.09874