Doclaynet Layout Eval

doclaynet-layout-eval

DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis — Pfitzmann et al. (2022) (arXiv:2206.01062, 2022)

What this evaluates

Evaluates the ability of object detection models to accurately identify and localize 11 distinct document layout elements (e.g., text, tables, figures, headers) on scanned or digital document pages. It measures robustness across diverse, real-world document types and tests how data splitting strategies and label definitions impact prediction accuracy.

Datasets

• DocLayNet — total 80863; splits: train (-1), val (-1), test (-1)

Metrics

• mAP@0.5-0.95 (primary) — range: [0, 1]
  • Mean Average Precision computed across 10 IoU thresholds ranging from 0.5 to 0.95 in steps of 0.05. Calculated using the official COCO API evaluation code.

Input / output format

Input: RGB document page images resized to 1025×1025 pixels.

Output: Bounding box coordinates (x, y, width, height) and class labels for each detected layout element.

Scoring recipe

coco_gt = COCO()  # Load ground-truth annotations in COCO format
coco_dt = coco_gt.loadRes(predictions)  # Load model predictions
coco_eval = COCOeval(coco_gt, coco_dt, 'bbox')
coco_eval.params.iouThrs = np.arange(0.50, 0.96, 0.05)
coco_eval.evaluate()
coco_eval.accumulate()
mAP = coco_eval.summarize()  # Returns average precision over IoU range

Common pitfalls

• Using a random page-wise train/test split instead of a document-wise split artificially inflates mAP by ~10% due to style leakage across splits.
• Merging or down-mapping layout classes (e.g., grouping paragraphs or dropping headers) before training significantly alters mAP scores and complicates cross-dataset comparison.
• Evaluating on different label sets without standardizing class mappings leads to incomparable mAP results across datasets like PubLayNet and DocBank.

Evidence (verbatim from paper)

Similarly as in PubLayNet, we will evaluate the quality of their predictions using mean average precision (mAP) with 10 overlaps that range from 0.5 to 0.95 in steps of 0.05 (mAP@0.5-0.95). These scores are computed by leveraging the evaluation code provided by the COCO API.

Citation

@misc{pfitzmann2022doclaynet,
  title={DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis},
  author={Pfitzmann et al. (2022)},
  year={2022},
  note={arXiv:2206.01062}
}

• arXiv: 2206.01062