Document Ie Eval

document-ie-eval

Information Redundancy and Biases in Public Document Information Extraction Benchmarks — Laatiri et al. (2023) (arXiv:2304.14936, 2023)

What this evaluates

Evaluates the ability of models to extract key information fields from visually-rich documents (receipts and forms). It probes generalization to unseen document templates by comparing performance on original versus resampled test splits.

Datasets

• SROIE — total ?; splits: test (-1), train (-1), val (-1)
• FUNSD — total ?; splits: test (-1), train (-1), val (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision is the ratio of correctly predicted entities to all predicted entities, and Recall is the ratio of correctly predicted entities to all actual entities.

Input / output format

Input: Visually-rich document images (receipts for SROIE, forms for FUNSD) with associated text and layout/positional information.

Output: Extracted key information fields (entities) from the document, typically represented as text spans or structured key-value pairs.

Scoring recipe

def compute_f1(predictions, gold):
    tp = len(set(predictions) & set(gold))
    fp = len(set(predictions) - set(gold))
    fn = len(set(gold) - set(predictions))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0

Common pitfalls

• Original test splits contain high template replication (75% for SROIE, 16% for FUNSD), allowing models to memorize templates rather than generalize.
• Performance drops significantly on resampled splits, especially for textual-only models, indicating the original benchmarks overestimate generalization capabilities.

Evidence (verbatim from paper)

On receipt understanding, F1 scores drastically drop compared to results on the original split in table 1, BERT, AlBERT and RoBERTa drop on average 10.5 F1 points whereas multi-modal models drop only 7.5 F1 points on average.

Citation

@misc{laatiri2023informationredundancy,
  title={Information Redundancy and Biases in Public Document Information Extraction Benchmarks},
  author={Laatiri et al. (2023)},
  year={2023},
  note={arXiv:2304.14936}
}

• arXiv: 2304.14936