One Shot Doc Ie Eval

one-shot-doc-ie-eval

One-shot Information Extraction from Document Images using Neuro-Deductive Program Synthesis — Sunder et al. (2019) (arXiv:1906.02427, 2019)

What this evaluates

This benchmark evaluates a system's ability to perform one-shot information extraction from document images. It measures how accurately the model can extract specific entity values (e.g., dates, amounts, names) from unseen test documents after being shown only a single training example, with optional supplementary documents for refinement.

Datasets

• Doctor's Bills — total 90; splits: train (10), test (80)
• Patent (Ghega) — total 136; splits: train (-1), test (-1)

Metrics

• extraction accuracy (primary) — range: percent
  • Accuracy is computed as the percentage of correctly extracted entity values across all test documents for a given entity type. The paper reports the average extraction accuracy per entity.

Input / output format

Input: A document image (I_test), a set of pre-synthesized extraction programs (P) for the document template, and the target entity name (f_i). The system first runs a Vision API on the image to populate a relational database.

Output: A single extracted entity value (v_i^test) corresponding to the target entity f_i.

Scoring recipe

def compute_extraction_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100
# Applied per entity type across all test documents

Common pitfalls

• The evaluation is strictly one-shot (1 training document), with the TrainNS variant allowed up to 2 supplementary documents. Readers may overlook this constraint when comparing to multi-shot baselines.
• The Patent dataset baseline from Ref. [13] solves a different task (bounding box detection) rather than exact value extraction, making direct accuracy comparisons invalid.
• Accuracy is averaged per entity type, not globally across all entities or documents, which can mask performance on specific fields.

Evidence (verbatim from paper)

We select a document from the training pool for one-shot learning and the rest for use as supplementary documents for running TrainNS algorithm. This is done multiple times such that each document in the training pool is used in one-shot learning once. The average extraction accuracy for every entity is reported in table 2.

Citation

@misc{sunder2019onedoc,
  title={One-shot Information Extraction from Document Images using Neuro-Deductive Program Synthesis},
  author={Sunder et al. (2019)},
  year={2019},
  note={arXiv:1906.02427}
}

• arXiv: 1906.02427