Document Parsing Eval

document-parsing-eval

Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training — Li et al. (2026) (arXiv:2603.23885, 2026)

What this evaluates

Evaluates end-to-end document parsing models on their ability to extract structured content (text, formulas, tables, reading order) from both standardized printed documents and real-world captured images. It measures structural fidelity, multilingual robustness, and decoding stability under visual degradation.

Datasets

• OmniDocBench — total ?; splits: test (-1)
• XFUND — total ?; splits: test (-1)
• Wild-OmniDocBench — total ?; splits: test (-1)

Metrics

• Overall (primary) — range: percent
  • Composite score aggregating performance across text, formula, table, and reading-order sub-metrics. Higher is better.
• TextEdit — range: other
  • Edit distance for extracted text content. Lower is better.
• FormulaCDM — range: percent
  • Character/Token accuracy for mathematical formulas. Higher is better.
• TableTEDS — range: percent
  • Tree Edit Distance based Similarity for table structure. Higher is better.
• Reading OrderEdit — range: other
  • Edit distance for reading order of document elements. Lower is better.
• Repeat — range: [0, 1]
  • Fraction of outputs containing an identical structured pattern repeated more than 10 times and reaching maximum generation length. Lower is better.

Input / output format

Input: Document images (scanned or digital) with varying layouts, illumination, and geometric distortions.

Output: Structured document representation (e.g., LaTeX or tokenized structure) generated autoregressively.

Scoring recipe

def compute_metrics(predictions, gold):
    text_edit = edit_distance(predictions.text, gold.text)
    formula_acc = token_accuracy(predictions.formulas, gold.formulas)
    table_teds = tree_edit_similarity(predictions.tables, gold.tables)
    order_edit = edit_distance(predictions.reading_order, gold.reading_order)
    overall = aggregate(text_edit, formula_acc, table_teds, order_edit)
    repeat_rate = sum(1 for p in predictions if is_repetitive(p)) / len(predictions)
    return {'TextEdit': text_edit, 'FormulaCDM': formula_acc, 'TableTEDS': table_teds, 'Reading OrderEdit': order_edit, 'Overall': overall, 'Repeat': repeat_rate}

Common pitfalls

• TextEdit and Reading OrderEdit are error metrics where lower values indicate better performance, opposite to accuracy-based metrics.
• The 'Overall' score is a composite aggregate rather than a single standardized metric; direct comparison requires checking the aggregation method.
• Repetition rate is strictly defined by pattern repetition >10 times AND reaching max generation length, not just any repeated text.

Evidence (verbatim from paper)

Comparing #1 with #4, our method delivers clear gains on both settings: +3.79 Overall on OmniDocBench and +8.21 on Wild-OmniDocBench, while the repetition rate drops from 4.7→2.1 and 8.6→4.3, respectively.

Citation

@misc{li2026dochumming,
  title={Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.23885}
}

• arXiv: 2603.23885