# Comp Hrdoc Eval

> Evaluates a model's ability to perform comprehensive hierarchical document structure analysis, including detecting page objects, predicting reading order across multiple groups, extracting tables of contents, and reconstructing the overall document hierarchy. Use when the user wants to benchmark on Comp-HRDoc, PubLayNet, DocLayNet, HRDoc, or asks about evaluating this task. Reports segmentation-based mAP.

- Skill: `qhjqhj00/comp-hrdoc-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/comp-hrdoc-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/comp-hrdoc-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/comp-hrdoc-eval

---


# comp-hrdoc-eval

> Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis — Wang et al. (2024) (arXiv:2401.11874, 2024)

## What this evaluates

Evaluates a model's ability to perform comprehensive hierarchical document structure analysis, including detecting page objects, predicting reading order across multiple groups, extracting tables of contents, and reconstructing the overall document hierarchy.

## Datasets

- **Comp-HRDoc** — total 1500; splits: train (1000), test (500); repo https://github.com/microsoft/CompHRDoc
- **PubLayNet** — total 364232; splits: train (340391), val (11858), test (11983)
- **DocLayNet** — total 80863; splits: train (69375), test (4999), val (6489)
- **HRDoc** — total 2500; splits: HRDS (1000), HRDH (1500)

## Metrics

- `segmentation-based mAP` **(primary)** — range: [0, 1]
  - COCO-style mean average precision computed over segmentation masks rather than bounding boxes, averaged across IoU thresholds from 0.50 to 0.95 in steps of 0.05.
- `REDS` — range: [0, 1]
  - Reading Edit Distance Score defined as 1 - D/N, where D is the minimum total Levenshtein distance between predicted and ground-truth reading order groups (matched via Hungarian algorithm), and N is the total number of basic units (text-lines and graphical objects).
- `Semantic-TEDS` — range: [0, 1]
  - Tree-edit-distance-based score for evaluating hierarchical structure reconstruction and table of contents extraction, measuring the similarity between predicted and ground-truth document trees.

## Input / output format

**Input**: Multi-page document images containing text and graphical elements.

**Output**: Predicted segmentation masks for page objects, ordered sequences of text-lines and graphical objects grouped by reading order (with paragraph ending markers), extracted table of contents, and a hierarchical tree structure representing document relations.

## Scoring recipe

```python
def compute_reds(pred_groups, gt_groups):
    D = hungarian_match(pred_groups, gt_groups, cost_fn=levenshtein_distance)
    N = count_basic_units(gt_groups)
    return 1 - (D / N)

def compute_mAP(pred_masks, gt_masks):
    return coco_compute_mAP(pred_masks, gt_masks, iou_range=(0.50, 0.95, 0.05))

def compute_teds(pred_tree, gt_tree):
    return tree_edit_distance_score(pred_tree, gt_tree)
```

## Common pitfalls

- Using bounding-box mAP instead of segmentation-based mAP for paragraphs, which can span multiple columns and require union-of-segmentations.
- Ignoring multiple independent reading order groups (e.g., multiple articles in a newspaper) and applying full-ranking metrics that assume a single global order.
- Neglecting paragraph segmentation errors when evaluating reading order, as previous metrics only ranked text-lines without accounting for Detect-stage segmentation mistakes.

## Evidence (verbatim from paper)

> In the page object detection task, we utilize the COCO-style segmentation-based mean average precision (mAP) evaluation metric rather than a box-based metric. ... Regarding the reading order prediction task, ... we propose a reading edit distance score (REDS) to evaluate the reading order task. ... Ultimately, we define 1 - D/N as the evaluation score for the reading order prediction task. For both the table of contents extraction and hierarchical structure reconstruction tasks, we opt for the Semantic-TEDS [7] as their evaluation metric.

## Citation

```bibtex
@misc{wang2024detectorderconstruct,
  title={Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.11874}
}
```

- arXiv: 2401.11874