Docile Eval

docile-eval

DocILE Benchmark for Document Information Localization and Extraction — Štepan Šimsa et al. (2023) (arXiv:2302.05658, 2023)

What this evaluates

Evaluates a model's ability to localize and extract key information (KILE) and recognize line items (LIR) in business documents. It probes multi-modal document understanding, specifically token-level classification and bounding box merging based on OCR and layout features.

Datasets

• DocILE — total 106700; splits: test (-1); repo https://github.com/rossumai/docile

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall computed over all predicted token-class assignments using a standard BIO tagging scheme. Matches are determined by overlapping predicted and ground-truth bounding boxes or token sequences.
• AP — range: [0, 1]
  • Average Precision computed from the precision-recall curve across different confidence thresholds for the multi-label classification task.
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive token-class assignments to the total number of predicted positive assignments.
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive token-class assignments to the total number of actual positive assignments in the ground truth.

Input / output format

Input: Document images paired with OCR-extracted tokens, including their bounding boxes and text content. Tokens are pre-ordered in top-down, left-to-right reading order per text line.

Output: For each OCR token, a predicted class label from the 55 fine-grained KILE/LIR classes using a BIO tagging scheme, plus line-item boundary tags (B-I, I-I, O-I, E-I). Predictions are merged horizontally/vertically based on spatial proximity thresholds to form final text blocks and bounding boxes.

Scoring recipe

preds = model.predict(doc_image, ocr_tokens)
gold = doc.gold_annotations

true_pos = 0
false_pos = 0
false_neg = 0

for pred_group in preds:
    matched = False
    for gold_group in gold:
        if match(pred_group, gold_group, threshold=0.5):
            true_pos += 1
            matched = True
            break
    if not matched:
        false_pos += 1

for gold_group in gold:
    if not any(match(pred_group, gold_group) for pred_group in preds):
        false_neg += 1

precision = true_pos / (true_pos + false_pos) if (true_pos + false_pos) > 0 else 0
recall = true_pos / (true_pos + false_neg) if (true_pos + false_neg) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• OCR tokens must be reordered in top-down, left-to-right order before processing; skipping this degrades performance significantly.
• Pre-training on external document datasets (e.g., IIT-CDIP) is strictly prohibited in the official benchmark, though some baselines may violate this.
• The merging strategy for final predictions is simplistic and relies on fixed spatial thresholds (e.g., 25% margin on text block height), which may not generalize to all layouts.

Evidence (verbatim from paper)

The primary metric for each task is shown in bold. Table 3: Baseline results for KILE & LIR. LayoutLMv3BASE, achieving the best results, was pre-trained on another document dataset - IIT-CDIP [37], which is prohibited in the official benchmark. The best results among permitted models are underlined. The primary metric for each task is shown in bold.

Citation

@misc{simsa2023docile,
  title={DocILE Benchmark for Document Information Localization and Extraction},
  author={Štepan Šimsa et al. (2023)},
  year={2023},
  note={arXiv:2302.05658}
}

• arXiv: 2302.05658