Visual Information Extraction Eval

visual-information-extraction-eval

Modeling Entities as Semantic Points for Visual Information Extraction in the Wild — Yang et al. (2023) (arXiv:2303.13095, 2023)

What this evaluates

Evaluates a model's ability to extract entity spans and link them to key-value pairs from complex, real-world document images. It probes joint vision-language understanding, handling poor image quality, occlusion, and multi-lingual text without relying on external OCR pipelines.

Datasets

• FUNSD — total 199; splits: train (149), test (50)
• XFUND — total ?; splits: train (-1), test (-1)
• CORD — total 1000; splits: train (800), val (100), test (100)
• SIBR — total 1000; splits: test (1000)

Metrics

• F1-score (primary) — range: percent
  • Harmonic mean of precision and recall for Entity Extraction (EE) and Entity Linking (EL). Calculated as 2 * (precision * recall) / (precision + recall). Precision and recall are computed by matching predicted boxes, categories, and links against ground truth annotations.

Input / output format

Input: Document images. Depending on the evaluation setting, inputs may include ground-truth or predicted text bounding boxes and OCR text contents.

Output: Predicted bounding boxes for entities, their categorical labels (for EE), and key-value pair links (for EL).

Scoring recipe

# For FUNSD/XFUND/CORD: aggregate multi-segment predictions into single entities
preds = aggregate_segments(predictions)
gold = aggregate_segments(gold)
# Match predicted boxes/labels/links to ground truth
tp_ee, fp_ee, fn_ee = match_entities(preds, gold, task="EE")
tp_el, fp_el, fn_el = match_links(preds, gold, task="EL")
# Compute F1 for each task
f1_ee = 2 * tp_ee / (2 * tp_ee + fp_ee + fn_ee)
f1_el = 2 * tp_el / (2 * tp_el + fp_el + fn_el)
return f1_ee, f1_el

Common pitfalls

• Comparing against LayoutLMv3 is unfair because it uses ground-truth text contents during evaluation, while the proposed method does not.
• On SIBR, the evaluation is strictly end-to-end; using external OCR or ground-truth annotations during inference will invalidate the reported F1 scores.
• Entity segments spanning multiple text lines must be aggregated into a single entity before matching to avoid undercounting.

Evidence (verbatim from paper)

Following [19], we aggregate segment features of the text into one entity if the entity has multiple segments on FUNSD, XFUND, and CORD. But when testing end-to-end on SIBR, model inference does not rely on any annotation information. After decoding, the boxes, categories, and links are directly matched with GT. We use F1-score as our evaluation metrics for both EE and EL tasks.

Citation

@misc{yang2023modeling,
  title={Modeling Entities as Semantic Points for Visual Information Extraction in the Wild},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2303.13095}
}

• arXiv: 2303.13095