qhjqhj00 doc-key-info-extraction-eval
Spatial Dual-Modality Graph Reasoning for Key Information Extraction — Hongbin Sun et al. (arXiv:2103.14470, 2021)
What this evaluates
Evaluates a model's ability to extract fine-grained key information categories (e.g., total price, date, address) from unstructured, template-agnostic document images. It probes robustness to spatial layout variations, dual-modality feature fusion, and generalization to unseen templates and OCR noise.
Datasets
- SROIE — total ?; splits: (unstated)
- WildReceipt — total ?; splits: (unstated)
Metrics
F1 score(primary) — range: percent- Harmonic mean of precision and recall for key information category prediction. Calculated per category and averaged across 12 value categories on WildReceipt, or reported as a single overall score on SROIE. Values are expressed as percentages.
Input / output format
Input: Document images resized to 512x512, accompanied by text bounding boxes and recognized text strings (either ground truth or OCR-generated). Visual features are extracted via U-Net, and textual features via Bi-LSTM.
Output: Predicted key information category label for each text region/node in the document graph (e.g., 'Total', 'Tax', 'Date', 'Prod item').
Scoring recipe
def compute_f1(predictions, gold):
tp = sum(1 for p, g in zip(predictions, gold) if p == g)
fp = sum(1 for p, g in zip(predictions, gold) if p != g)
fn = sum(1 for p, g in zip(predictions, gold) if p != g)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1 * 100
Common pitfalls
- Performance drops significantly (~6.3%) when using OCR-recognized text instead of ground truth text due to character misrecognition.
- Mismatch between detected and ground truth text boxes (e.g., 1-to-many overlaps) introduces noisy signals if matched by maximum IoU alone.
- Ablation shows removing graph reasoning causes an 11.5% F1 drop, highlighting that spatial message passing is critical for this task.
Evidence (verbatim from paper)
Specifically, SDMG-R achieves 11.8%, 9.7%, and 3.0% absolute improvements in terms of $F_{1}$ score averaged on 12 value categories on WildReceipt compared with Chargrid, Chargrid-UNet, and VRD respectively.
Citation
@misc{sun2021spatial,
title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction},
author={Hongbin Sun et al.},
year={2021},
note={arXiv:2103.14470}
}
- arXiv: 2103.14470