Docile 2023 Eval

docile-2023-eval

DocILE 2023 Teaser: Document Information Localization and Extraction — Štěpán Šimsa et al. (2023) (arXiv:2301.12394, 2023)

What this evaluates

Evaluates a model's ability to localize and extract key information fields and line items from diverse business documents. It specifically probes spatial grounding of text values against predefined field types and tests generalization to previously unseen document layouts.

Datasets

• DocILE — total 6680; splits: train (-1), val (-1), test (-1)

Metrics

• Average Precision (PCC-based) (primary) — range: [0, 1]
  • Standard Average Precision computed by matching predicted bounding boxes to ground truth fields. A prediction is a true positive only if its bounding box contains exactly the set of Pseudo-Character Centers (PCCs) defined for that field, rather than using standard IoU thresholding.
• Micro F1 (Line Item) — range: [0, 1]
  • Micro F1 score over all line item fields. A field is correct if it satisfies the PCC-based localization criteria and is assigned to the correct line item. Matching between predicted and ground truth line items is solved via maximum matching that maximizes overall recall.

Input / output format

Input: Pre-processed document PDFs (normalized to 150 DPI) with optional OCR token detections and locations. Ground truth provides field types, bounding boxes, and line item groupings.

Output: Predicted bounding boxes for each key information field (with field type), and optionally line item groupings. Boxes must align with Pseudo-Character Centers.

Scoring recipe

def score_track1(predictions, ground_truth):
    tp, fp, fn = 0, 0, 0
    used_preds = set()
    for gt in ground_truth:
        gt_pccs = get_pccs(gt.box)
        best_match = None
        for i, pred in enumerate(predictions):
            if i in used_preds: continue
            pred_pccs = get_pccs(pred.box)
            if pred_pccs == gt_pccs:
                best_match = i
                break
        if best_match is not None:
            tp += 1
            used_preds.add(best_match)
        else:
            fn += 1
    fp = len(predictions) - len(used_preds)
    return compute_ap(tp, fp, fn)

def score_track2(predictions, ground_truth):
    # Group fields into line items, then apply max matching to maximize recall
    # Compute micro F1 over all correctly matched fields

Common pitfalls

• Using standard IoU thresholding for bounding box matching instead of the required exact Pseudo-Character Center (PCC) containment rule.
• Evaluating text recognition accuracy as part of the primary leaderboard metric; text extraction is secondary and not required for challenge submissions.
• Ignoring the layout generalization constraint: val/test sets are explicitly split to contain unseen layouts, so models trained only on seen layouts will fail.

Evidence (verbatim from paper)

Since the task is framed as a detection problem, the standard Average Precision metric will be used as the main evaluation metric. Unlike the common practice in object detection, where true positives are determined by thresholding the Intersection-over-Union, we use a different criterion tailored to better evaluate the usefulness of detections for text read-out. Inspired by the CLEval metric [[1]] used in text detection, we measure whether the predicted area contains all related character centers (and none others).

Citation

@misc{simsa2023docile,
  title={DocILE 2023 Teaser: Document Information Localization and Extraction},
  author={Štěpán Šimsa et al. (2023)},
  year={2023},
  note={arXiv:2301.12394}
}

• arXiv: 2301.12394