Docvqa A Dataset For Vqa On Document Images Arxiv 2007 00398v3

DocVQA: A Dataset for VQA on Document Images

One-line decision

Use this skill when you need a VQA dataset on document images for training and evaluating document understanding VLMs. Avoid it when you need scene-level VQA rather than document understanding.

Skill metadata

• Skill type: document-vqa-dataset
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Create a VQA dataset on document images where questions require understanding document layout, text content, and structure.

Problem signature

• Modality: document images (invoices, forms, reports, etc.) with questions about their content.
• Data state: 12.7K document images with 50K human-annotated QA pairs.
• Scale regime: 50K QA pairs on 12.7K document images.
• Model requirement: No model required for dataset construction; validated with document understanding models.

Use when

• You need document VQA training or evaluation data.
• You want to benchmark VLM document understanding.
• You need questions about document layout and structured content.

Do not use when

• You need scene-level VQA.
• Document understanding is not your target task.
• You need a much larger dataset.

Required inputs

• document_images: Diverse document images (invoices, forms, reports, letters, etc.).
• human_annotators: Workers to write questions about document content.
• answer_annotations: Ground truth answers with evidence spans.

Optional inputs

• ocr_output: OCR text and layout information for the documents.

Outputs

• docvqa_dataset: 50K QA pairs on 12.7K document images.
• evaluation_metrics: ANLS (Average Normalized Levenshtein Similarity) metric.

Assumptions and prerequisites

• Document understanding requires specialized VQA data beyond scene-level datasets.
• Questions about documents require layout and structure understanding.
• The ANLS metric better evaluates document VQA than exact match.

Procedure

1. Collect diverse document images Action: Gather documents from various domains (industry, government, academic). Why: Diversity ensures broad document understanding. Note: See paper for details.
2. Generate QA pairs Action: Have annotators create questions answerable from the document. Why: Human questions ensure relevance and difficulty. Note: See paper for details.
3. Annotate evidence Action: Mark the evidence region in the document for each answer. Why: Evidence annotation enables interpretable evaluation. Note: See paper for details.
4. Define evaluation metric Action: Use ANLS instead of exact match for flexible evaluation. Why: ANLS accounts for minor OCR and transcription errors. Note: See paper for details.

Parameters to set

• document_types — Role: Types of documents included. How to set: Include invoices, forms, reports, letters, scientific papers. Default/range: Diverse. Effect: More types improve generalization.
• questions_per_doc — Role: Average questions per document. How to set: 3-5 questions per document. Default/range: ~4. Effect: More questions increase dataset utility.

Validation checks

• Questions should require understanding document structure, not just text.
• The ANLS metric should handle OCR noise appropriately.
• Models should need both text reading and layout understanding.

Failure modes

• Some questions may be answerable from text alone without layout understanding.
• Document quality variation may affect recognition.
• The dataset may not cover all document types.

Adaptation notes for VLM training

• DocVQA is a standard benchmark and training data for document VLMs.
• Include DocVQA in instruction tuning for document understanding capability.
• The ANLS metric is now used across document understanding benchmarks.

Implementation notes

• Provide OCR and layout information alongside images.
• Use the ANLS metric for evaluation.
• Track performance by document type.

Evidence from the paper

• DocVQA provides 50K QA pairs on 12.7K document images across diverse document types.
• The dataset requires understanding document layout and structure, not just text.
• The ANLS metric provides flexible evaluation accounting for OCR noise.
• DocVQA has become a standard benchmark for document understanding VLMs.

Source paper

• Title: DocVQA: A Dataset for VQA on Document Images
• Year: 2021
• Venue: WACV
• Paper ID: arxiv-2007.00398v3
• URL: http://arxiv.org/abs/2007.00398v3
• arXiv ID: 2007.00398v3