Icdar2019 Sroie Eval

icdar2019-sroie-eval

ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction — Huang et al. (2021) (arXiv:2103.10213, 2021)

What this evaluates

Evaluates end-to-end document understanding on low-quality scanned receipts, specifically testing text localization, character-level OCR, and structured key information extraction (e.g., company, cash, date, address).

Datasets

• ICDAR2019 SROIE — total 1000; splits: train (-1), test (-1)

Metrics

• primary metric (primary) — range: percent
  • The competition uses a distinct primary metric for each of the three tasks. Task 1 evaluates bounding box detection accuracy. Task 2 evaluates character/word recognition accuracy. Task 3 evaluates exact match of extracted key fields (company, cash, date, address) against ground truth. Exact formulas are not specified in the provided section.

Input / output format

Input: Scanned receipt images characterized by low quality, complex layouts, folds, poor ink, and small fonts.

Output: Task 1: Bounding boxes for text lines. Task 2: Recognized text content per line. Task 3: Structured JSON fields for key information (company, cash, date, address).

Scoring recipe

for each task in [1, 2, 3]:
  predictions = load_submission(task)
  gold = load_ground_truth(task)
  score = compute_primary_metric(predictions, gold)
  # Task 1: Bounding box overlap/detection
  # Task 2: Character/word recognition accuracy
  # Task 3: Exact match of key fields (company, cash, date, address)
  rankings.append((team, score))
winners = sort_by_score(rankings, descending=True)

Common pitfalls

• Low-quality conditions (folded receipts, poor ink, small fonts) significantly degrade baseline performance.
• Task 3 requires layout-aware extraction; methods relying solely on OCR without spatial/contextual constraints often fail.
• Ensemble methods and post-processing (e.g., regex, lexicon correction) are critical for competitive scores.

Evidence (verbatim from paper)

After the submission deadlines, we collected all submissions and evaluate their performance through automated process with scripts developed by the RRC web team. The winners are determined for each task based on the score achieved by the corresponding primary metric.

Citation

@misc{huang2021icdar2019sroie,
  title={ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction},
  author={Huang et al. (2021)},
  year={2021},
  note={arXiv:2103.10213}
}

• arXiv: 2103.10213