# Doc Key Info Extraction Eval

> Evaluates a model's ability to extract fine-grained key information categories (e.g., total price, date, address) from unstructured, template-agnostic document images. It probes robustness to spatial layout variations, dual-modality feature fusion, and generalization to unseen templates and OCR noise. Use when the user wants to benchmark on SROIE, WildReceipt, or asks about evaluating this task. Reports F1 score.

- Skill: `qhjqhj00/doc-key-info-extraction-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/doc-key-info-extraction-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/doc-key-info-extraction-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/doc-key-info-extraction-eval

---


# doc-key-info-extraction-eval

> Spatial Dual-Modality Graph Reasoning for Key Information Extraction — Hongbin Sun et al. (arXiv:2103.14470, 2021)

## What this evaluates

Evaluates a model's ability to extract fine-grained key information categories (e.g., total price, date, address) from unstructured, template-agnostic document images. It probes robustness to spatial layout variations, dual-modality feature fusion, and generalization to unseen templates and OCR noise.

## Datasets

- **SROIE** — total ?; splits: (unstated)
- **WildReceipt** — total ?; splits: (unstated)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Harmonic mean of precision and recall for key information category prediction. Calculated per category and averaged across 12 value categories on WildReceipt, or reported as a single overall score on SROIE. Values are expressed as percentages.

## Input / output format

**Input**: Document images resized to 512x512, accompanied by text bounding boxes and recognized text strings (either ground truth or OCR-generated). Visual features are extracted via U-Net, and textual features via Bi-LSTM.

**Output**: Predicted key information category label for each text region/node in the document graph (e.g., 'Total', 'Tax', 'Date', 'Prod item').

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    fp = sum(1 for p, g in zip(predictions, gold) if p != g)
    fn = sum(1 for p, g in zip(predictions, gold) if p != g)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Performance drops significantly (~6.3%) when using OCR-recognized text instead of ground truth text due to character misrecognition.
- Mismatch between detected and ground truth text boxes (e.g., 1-to-many overlaps) introduces noisy signals if matched by maximum IoU alone.
- Ablation shows removing graph reasoning causes an 11.5% F1 drop, highlighting that spatial message passing is critical for this task.

## Evidence (verbatim from paper)

> Specifically, SDMG-R achieves 11.8%, 9.7%, and 3.0% absolute improvements in terms of $F_{1}$ score averaged on 12 value categories on WildReceipt compared with Chargrid, Chargrid-UNet, and VRD respectively.

## Citation

```bibtex
@misc{sun2021spatial,
  title={Spatial Dual-Modality Graph Reasoning for Key Information Extraction},
  author={Hongbin Sun et al.},
  year={2021},
  note={arXiv:2103.14470}
}
```

- arXiv: 2103.14470