Smoldocling Doc Eval

smoldocling-doc-eval

SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion — Ahmed Nassar et al. (2025) (arXiv:2503.11576, 2025)

What this evaluates

This evaluation probes a vision-language model's ability to perform end-to-end document conversion, including text recognition, layout analysis, table and chart structure extraction, and code/formula parsing. It measures how accurately the model reconstructs document content and spatial structure from page images into standardized markup formats.

Datasets

• DocLayNet — total ?; splits: test (-1)
• SynthCodeNet — total ?; splits: test (-1)
• Im2Latex-230k — total ?; splits: test (-1)
• FinTabNet — total ?; splits: test (-1)
• PubTables-1M — total ?; splits: test (-1)

Metrics

• text-similarity metric — range: [0, 1]
  • Measures character or string similarity between normalized plain-text or LaTeX outputs. Requires formatting without non-textual elements.
• mAP@0.5:0.95 (primary) — range: [0, 1]
  • Mean Average Precision for bounding box localization across six layout classes (Text, Section Heading, List Item, Table, Picture, Formula). Computed over IoU thresholds from 0.5 to 0.95.
• TEDS (primary) — range: [0, 1]
  • Tree Edit Distance Similarity. Measures the structural similarity between predicted and ground-truth table or chart HTML trees, optionally including or excluding text content.

Input / output format

Input: Page images standardized at 144 DPI, processed in batches of up to three pages. Maximum sequence length of 8,192 tokens.

Output: A sequence of DocTags markup encoding content, structure, and layout (14 distinct layout-class tags). For evaluation, outputs are converted to Markdown, plaintext, LaTeX, or HTML depending on the task.

Scoring recipe

def score_ocr(pred, gold):
    pred_fmt = convert_to_markdown_or_latex(pred)
    gold_fmt = normalize(gold)
    return text_similarity_metric(pred_fmt, gold_fmt)

def score_layout(pred_boxes, pred_labels, gold_boxes, gold_labels):
    classes = ['Text', 'Section Heading', 'List Item', 'Table', 'Picture', 'Formula']
    pred_labels = map_to_classes(pred_labels, classes)
    gold_labels = map_to_classes(gold_labels, classes)
    return compute_mAP(pred_boxes, pred_labels, gold_boxes, gold_labels, iou_thresh=[0.5, 0.95])

def score_tables_charts(pred_html, gold_html):
    pred_tree = parse_html_to_tree(pred_html)
    gold_tree = parse_html_to_tree(gold_html)
    return 1.0 - (tree_edit_distance(pred_tree, gold_tree) / max(len(pred_tree), len(gold_tree)))

Common pitfalls

• Input resolution varies across literature; this protocol standardizes at 144 DPI to ensure fair comparison.
• Output markup conventions differ (Markdown, HTML, DocTags), requiring explicit normalization and class mapping before scoring.
• Low-resolution image crops (72 DPI) in FinTabNet/PubTables-1M degrade text transcription, artificially lowering TEDS scores that include text content.
• Table row/column spans are lost when converting to Markdown, necessitating separate HTML-based evaluation for accurate structure scoring.

Evidence (verbatim from paper)

Table Structure Recognition. We evaluate the accuracy of recovering table structures, i.e. shaping columns and rows with the correct text content and spans, through the TEDS metric[[104]], once including table text content and once on structure only with ommitted text (see Table[4]).

Citation

@misc{nassar2025smoldocling,
  title={SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion},
  author={Ahmed Nassar et al. (2025)},
  year={2025},
  note={arXiv:2503.11576}
}

• arXiv: 2503.11576