Receiptsense Eval

receiptsense-eval

ReceiptSense: Beyond Traditional OCR -- A Dataset for Receipt Understanding — Abdelrahman Abdallah et al. (2024) (arXiv:2406.04493, 2024)

What this evaluates

Evaluates multilingual receipt understanding across four tasks: question answering, object detection/classification, OCR, and information extraction. Probes model capabilities in handling real-world noise, mixed Arabic-English layouts, and complex formatting.

Datasets

• ReceiptSense — total ?; splits: test (-1); repo https://github.com/Update-For-Integrated-Business-AI/CORU

Metrics

• exact match (primary) — range: [0, 1]
  • Proportion of predictions that exactly match the ground truth answer or extracted field value.
• contains — range: [0, 1]
  • Proportion of predictions that contain the ground truth string as a substring.
• precision — range: [0, 1]
  • Proportion of predicted values that are correct relative to the total number of predictions.
• recall — range: [0, 1]
  • Proportion of ground truth values that are correctly predicted relative to the total number of ground truth instances.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• CER — range: percent
  • Character Error Rate: normalized Levenshtein distance between predicted and ground truth character sequences.
• WER — range: percent
  • Word Error Rate: normalized Levenshtein distance between predicted and ground truth word sequences.
• mAP50 — range: percent
  • Mean Average Precision at IoU threshold 0.50 across all object classes.
• mAP50-95 — range: percent
  • Mean Average Precision averaged over IoU thresholds from 0.50 to 0.95.

Input / output format

Input: Multilingual receipt images (Arabic-English) paired with questions or target information categories (e.g., Brand, Weight, #Units).

Output: Text answers for QA, bounding box coordinates and class labels for object detection, recognized character/word sequences for OCR, and extracted field values for information extraction.

Scoring recipe

def score(predictions, gold):
    exact_match = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    contains = sum(1 for p, g in zip(predictions, gold) if g in p) / len(gold)
    precision = sum(1 for p, g in zip(predictions, gold) if p in g) / len(predictions)
    recall = sum(1 for p, g in zip(predictions, gold) if p in g) / len(gold)
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    cer = sum(edit_distance(p, g) for p, g in zip(predictions, gold)) / sum(len(g) for g in gold)
    wer = sum(word_edit_distance(p, g) for p, g in zip(predictions, gold)) / sum(len(g.split()) for g in gold)
    return {'exact_match': exact_match, 'contains': contains, 'precision': precision, 'recall': recall, 'f1': f1, 'cer': cer, 'wer': wer}

Common pitfalls

• Zero-shot settings yield very low performance for categories like Brand and Weight, making few-shot evaluation essential for meaningful comparison.
• Models show high variance across different receipt fields (e.g., #Units vs. Pack), so overall F1 can mask category-specific failures.
• Traditional OCR baselines (Tesseract) struggle significantly with multilingual layouts, requiring specialized models for competitive CER/WER.

Evidence (verbatim from paper)

Figure[7] provides a comparative analysis of various large language models on the Receipt QA subset of ReceiptSense across four key metrics: precision, recall, exact match, and contains. GPT-4o consistently outperforms other models, achieving the highest scores across all metrics — notably 37.7% precision, 36.4% recall, 35.0% exact match, and 29.1% contains.

Citation

@misc{abdallah2024receiptsense,
  title={ReceiptSense: Beyond Traditional OCR -- A Dataset for Receipt Understanding},
  author={Abdelrahman Abdallah et al. (2024)},
  year={2024},
  note={arXiv:2406.04493}
}

• arXiv: 2406.04493