Omnidocbench Eval

omnidocbench-eval

MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm — Zhang Li et al. (2025) (arXiv:2506.05218, 2025)

What this evaluates

Evaluates end-to-end document parsing capabilities, including text recognition, formula recognition, table structure extraction, and reading order prediction across diverse document types and languages.

Datasets

• OmniDocBench — total 981; splits: test (981)

Metrics

• OverallEdit (primary) — range: [0, 1]
  • Character-level edit distance between the predicted and ground-truth parsed document text. Lower values indicate better alignment.
• TextEdit — range: [0, 1]
  • Character-level edit distance computed only on text blocks. Lower is better.
• FormulaEdit — range: [0, 1]
  • Character-level edit distance for formula regions. Lower is better.
• FormulaCDM — range: [0, 1]
  • Character-level Document Matching score measuring alignment accuracy for formulas. Higher is better.
• TableTEDS — range: [0, 1]
  • Tree Edit Distance based Similarity for table structure prediction. Higher is better.
• TableEdit — range: [0, 1]
  • Character-level edit distance for table content. Lower is better.
• Read OrderEdit — range: [0, 1]
  • Edit distance for the predicted reading order sequence compared to ground truth. Lower is better.

Input / output format

Input: Single PDF pages (images) containing mixed content (text, formulas, tables, figures).

Output: Parsed document structure including recognized text, LaTeX/formula representations, table structures, and reading order sequence.

Scoring recipe

def compute_metrics(pred, gold):
    # Edit distances (lower is better)
    overall_edit = levenshtein_distance(pred.text, gold.text)
    text_edit = levenshtein_distance(pred.text_blocks, gold.text_blocks)
    formula_edit = levenshtein_distance(pred.formulas, gold.formulas)
    read_order_edit = levenshtein_distance(pred.reading_order, gold.reading_order)
    table_edit = levenshtein_distance(pred.tables_content, gold.tables_content)
    # Structure/Alignment scores (higher is better)
    formula_cdm = character_document_matching(pred.formulas, gold.formulas)
    table_teds = tree_edit_distance_similarity(pred.table_structure, gold.table_structure)
    return {
        'OverallEdit': overall_edit,
        'TextEdit': text_edit,
        'FormulaEdit': formula_edit,
        'FormulaCDM': formula_cdm,
        'TableTEDS': table_teds,
        'TableEdit': table_edit,
        'Read OrderEdit': read_order_edit
    }

Common pitfalls

• Edit distance scores are highly sensitive to tokenization and formatting differences (e.g., spacing, line breaks), which can disproportionately penalize minor structural variations.
• Language-specific tokenization affects EN vs ZH scores; Chinese characters are typically evaluated at the character level, while English may use word-level tokenization, making cross-lingual comparison non-trivial.
• TableTEDS requires exact tree structure alignment; missing or merged cells in predictions drastically reduce scores even if visual layout appears correct.

Evidence (verbatim from paper)

OmniDocBench is a benchmark designed to evaluate real-world document parsing capabilities. It comprises 981 PDF pages spanning 9 document types, 4 layout styles, and 3 language categories. As shown in Table 2, MonkeyOCR achieves the best overall performance on both Chinese and English document parsing tasks. In particular, MonkeyOCR surpasses MinerU by over 6% in overall edit distance for Chinese documents, exceeds MinerU by an average of 15.0% in formula recognition across Chinese and English, and outperforms MinerU by 8.6% on average in table recognition for both languages.

Citation

@misc{zhang2025monkeyocr,
  title={MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm},
  author={Zhang Li et al. (2025)},
  year={2025},
  note={arXiv:2506.05218}
}

• arXiv: 2506.05218