Publaynet Layout Eval

publaynet-layout-eval

PubLayNet: largest dataset ever for document layout analysis — Zhong et al. (2019) (arXiv:1908.07836, 2019)

What this evaluates

Evaluates the ability of object detection models to identify and localize document layout elements (text, title, list, table, figure) in scientific PDF pages. It also probes transfer learning capabilities by fine-tuning on out-of-domain documents and table detection tasks.

Datasets

• PubLayNet — total ?; splits: dev (-1), test (-1); repo https://github.com/ibm-aur-nlp/PubLayNet

Metrics

• MAP @ IOU [0.50:0.95] (primary) — range: [0, 1]
  • Mean Average Precision averaged over Intersection over Union thresholds from 0.50 to 0.95 (step 0.05), following the COCO detection evaluation protocol.

Input / output format

Input: PDF pages converted to images.

Output: Bounding boxes with class labels for five categories: Text, Title, List, Table, Figure.

Scoring recipe

def compute_map(predictions, ground_truth):
    aps = []
    for iou_th in np.arange(0.50, 0.96, 0.05):
        tp, fp = 0, 0
        for gt in ground_truth:
            best_pred = max(predictions, key=lambda p: iou(p, gt))
            if iou(best_pred, gt) >= iou_th and not best_pred.used:
                tp += 1; best_pred.used = True
            else:
                fp += 1
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / len(ground_truth)
        aps.append(interpolate_ap(prec, rec))
    return np.mean(aps)

Common pitfalls

• Titles are consistently harder to detect than tables/figures due to smaller size and less distinctive shapes.
• Zero-shot transfer to out-of-domain documents (e.g., SPD) performs poorly; fine-tuning is required for comparable performance.
• Evaluation uses development and test sets, but exact split sizes are not provided in the paper.

Evidence (verbatim from paper)

The evaluation metric is the mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95] of bounding boxes, which is used in the COCO competition. Both models can generate accurate (MAP > 0.9) document layout, where M-RCNN shows a small advantage over F-RCNN.

Citation

@misc{zhong2019publaynet,
  title={PubLayNet: largest dataset ever for document layout analysis},
  author={Zhong et al. (2019)},
  year={2019},
  note={arXiv:1908.07836}
}

• arXiv: 1908.07836