Document Understanding Eval

document-understanding-eval

Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5 — Lamott et al. (2024) (arXiv:2409.11282, 2024)

What this evaluates

Evaluates a model's capability to perform document understanding tasks, including key information extraction, visual question answering, table question answering, and structural reading comprehension, using textual layout representations derived from OCR and spatial verbalization.

Datasets

• DocVQA — total ?; splits: train (-1), test (-1)
• InfographicsVQA — total ?; splits: train (-1), test (-1)
• WikiTableQuestions — total ?; splits: train (-1), test (-1)
• TabFact — total ?; splits: train (-1), test (-1)
• SROIE — total ?; splits: train (-1), test (-1)

Metrics

• ANLS (primary) — range: [0, 1]
  • Average Normalized Levenshtein Similarity; computes the normalized Levenshtein distance between predicted and ground truth text, averaged across samples.
• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy; proportion of samples where the model's prediction exactly matches the ground truth answer.
• type-aware accuracy — range: [0, 1]
  • Accuracy measure for SROIE that requires both the correct entity value and the correct entity type to be predicted.

Input / output format

Input: Textual layout representations generated via LAPDoc SpatialFormat, combined with OCR text (microsoft_cv or tesseract), formatted as task-specific prompts.

Output: JSON-formatted answers or extracted information matching the task instructions.

Scoring recipe

def score(predictions, gold, task):
    if task in ['DocVQA', 'InfographicsVQA']:
        return compute_anls(predictions, gold)
    elif task in ['TabFact', 'WikiTableQuestions']:
        return exact_match_accuracy(predictions, gold)
    elif task == 'SROIE':
        return type_aware_accuracy(predictions, gold)
    else:
        raise ValueError('Unknown task')

Common pitfalls

• Models frequently fail to follow task instructions and generate the requested JSON output format, resulting in 0.0 scores.
• Datasets with very high page counts (e.g., DeepForm, Kleister Charity, PWC) are explicitly excluded, so performance does not generalize to multi-page documents.
• Curriculum learning temperature schedules significantly impact convergence and final scores; improper scheduling can degrade performance.

Evidence (verbatim from paper)

The evaluation is performed following the procedure presented in Lamott et al.: For the DUE datasets, we use the official evaluation repository with its given metrics: ANLS for DocVQA and InfographicsVQA and accuracy for TabFact and WikiTableQuestions. For SROIE, we use a type-aware accuracy measure.

Citation

@misc{lamott2024leveraging,
  title={Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5},
  author={Lamott et al. (2024)},
  year={2024},
  note={arXiv:2409.11282}
}

• arXiv: 2409.11282