Comp Hrdoc Eval

comp-hrdoc-eval

Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis — Wang et al. (2024) (arXiv:2401.11874, 2024)

What this evaluates

Evaluates a model's ability to perform comprehensive hierarchical document structure analysis, including detecting page objects, predicting reading order across multiple groups, extracting tables of contents, and reconstructing the overall document hierarchy.

Datasets

• Comp-HRDoc — total 1500; splits: train (1000), test (500); repo https://github.com/microsoft/CompHRDoc
• PubLayNet — total 364232; splits: train (340391), val (11858), test (11983)
• DocLayNet — total 80863; splits: train (69375), test (4999), val (6489)
• HRDoc — total 2500; splits: HRDS (1000), HRDH (1500)

Metrics

• segmentation-based mAP (primary) — range: [0, 1]
  • COCO-style mean average precision computed over segmentation masks rather than bounding boxes, averaged across IoU thresholds from 0.50 to 0.95 in steps of 0.05.
• REDS — range: [0, 1]
  • Reading Edit Distance Score defined as 1 - D/N, where D is the minimum total Levenshtein distance between predicted and ground-truth reading order groups (matched via Hungarian algorithm), and N is the total number of basic units (text-lines and graphical objects).
• Semantic-TEDS — range: [0, 1]
  • Tree-edit-distance-based score for evaluating hierarchical structure reconstruction and table of contents extraction, measuring the similarity between predicted and ground-truth document trees.

Input / output format

Input: Multi-page document images containing text and graphical elements.

Output: Predicted segmentation masks for page objects, ordered sequences of text-lines and graphical objects grouped by reading order (with paragraph ending markers), extracted table of contents, and a hierarchical tree structure representing document relations.

Scoring recipe

def compute_reds(pred_groups, gt_groups):
    D = hungarian_match(pred_groups, gt_groups, cost_fn=levenshtein_distance)
    N = count_basic_units(gt_groups)
    return 1 - (D / N)

def compute_mAP(pred_masks, gt_masks):
    return coco_compute_mAP(pred_masks, gt_masks, iou_range=(0.50, 0.95, 0.05))

def compute_teds(pred_tree, gt_tree):
    return tree_edit_distance_score(pred_tree, gt_tree)

Common pitfalls

• Using bounding-box mAP instead of segmentation-based mAP for paragraphs, which can span multiple columns and require union-of-segmentations.
• Ignoring multiple independent reading order groups (e.g., multiple articles in a newspaper) and applying full-ranking metrics that assume a single global order.
• Neglecting paragraph segmentation errors when evaluating reading order, as previous metrics only ranked text-lines without accounting for Detect-stage segmentation mistakes.

Evidence (verbatim from paper)

In the page object detection task, we utilize the COCO-style segmentation-based mean average precision (mAP) evaluation metric rather than a box-based metric. ... Regarding the reading order prediction task, ... we propose a reading edit distance score (REDS) to evaluate the reading order task. ... Ultimately, we define 1 - D/N as the evaluation score for the reading order prediction task. For both the table of contents extraction and hierarchical structure reconstruction tasks, we opt for the Semantic-TEDS [7] as their evaluation metric.

Citation

@misc{wang2024detectorderconstruct,
  title={Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.11874}
}

• arXiv: 2401.11874