Docbank Layout Eval

docbank-layout-eval

DocBank: A Benchmark Dataset for Document Layout Analysis — Minghao Li et al. (2020) (arXiv:2006.01038, 2020)

What this evaluates

Evaluates a model's ability to identify and classify semantic document structures (e.g., sections, figures, equations) from serialized 2D document pages. It probes multimodal layout understanding by measuring how well token-level predictions align with ground-truth semantic units, even when tokens are discontinuous.

Datasets

• DocBank — total ?; splits: test (-1); repo https://github.com/doc-analysis/DocBank

Metrics

• F1 Score (primary) — range: [0, 1]
  • Precision = Area of Ground truth tokens in Detected tokens / Area of all Detected tokens. Recall = Area of Ground truth tokens in Detected tokens / Area of all Ground truth tokens. F1 Score = 2 * Precision * Recall / (Precision + Recall). Area is computed as the count of tokens in the respective sets.

Input / output format

Input: Serialized sequence of tokens from a 2D document page, truncated to a maximum block size of 512. Includes layout/position features.

Output: Token-level classification labels for 12 semantic structure categories (Abstract, Author, Caption, Equation, Figure, Footer, List, Paragraph, Reference, Section, Table, Title).

Scoring recipe

def compute_f1(pred_tokens, gt_tokens):
    intersection = len(set(pred_tokens) & set(gt_tokens))
    precision = intersection / len(pred_tokens) if pred_tokens else 0
    recall = intersection / len(gt_tokens) if gt_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
# Compute per category, then macro-average across all 12 semantic structures

Common pitfalls

• Standard BIO-tagging evaluation fails because tokens belonging to the same semantic unit can be discontinuous in the serialized sequence.
• Metrics are computed per semantic category, not as a single global accuracy; results must be macro-averaged across all 12 structure types.
• Vision models output bounding boxes, which must be converted to token-level labels before applying the token-area metric.

Evidence (verbatim from paper)

As the inputs of our model are serialized 2-D documents, the typical BIO-tagging evaluation is not suitable for our task. The tokens of each semantic unit may discontinuously distribute in the input sequence. In this case, we proposed a new metric, especially for text-based document layout analysis methods. For each kind of document semantic structure, we calculated their metrics individually. The definition is as follows: Precision = Area of Ground truth tokens in Detected tokens / Area of all Detected tokens, Recall = Area of Ground truth tokens in Detected tokens / Area of all Ground truth tokens, F1 Score = 2 * Precision * Recall / (Precision + Recall).

Citation

@misc{li2020docbank,
  title={DocBank: A Benchmark Dataset for Document Layout Analysis},
  author={Minghao Li et al. (2020)},
  year={2020},
  note={arXiv:2006.01038}
}

• arXiv: 2006.01038