Textvqa Towards Reasoning About Text In Images Arxiv 1904 08920v2

TextVQA: Towards Reasoning about Text in Images

One-line decision

Use this skill when you need a VQA dataset requiring models to read and reason about text visible in images. Avoid it when your VQA task does not involve reading text in images.

Skill metadata

• Skill type: text-rich-vqa-dataset
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Create a VQA dataset where answering questions requires reading and reasoning about text visible in images, addressing a gap in existing VQA benchmarks.

Problem signature

• Modality: images with visible text and questions requiring text reading and reasoning.
• Data state: images from OpenImages with human-written questions about visible text.
• Scale regime: 45K questions on 28K images from OpenImages.
• Model requirement: No model required for dataset construction; validated with VQA models + OCR.

Use when

• You need VQA data requiring text reading in images.
• You want to benchmark VLM OCR and text reasoning capabilities.
• You need training data for text-rich visual understanding.

Do not use when

• Your VQA task does not involve text in images.
• You need a larger-scale text-reading dataset.
• You need document-specific rather than scene text understanding.

Required inputs

• text_rich_images: Images containing visible text (signs, labels, screens, etc.).
• human_annotators: Workers to write questions requiring text reading.
• answer_annotations: Ground truth answers for each question.

Optional inputs

• ocr_tokens: OCR-extracted text to assist model development.

Outputs

• textvqa_dataset: 45K questions on 28K images requiring text reading.
• evaluation_metrics: VQA accuracy metrics for text-reading evaluation.

Assumptions and prerequisites

• Existing VQA datasets do not sufficiently test text reading in images.
• Questions requiring text reading are a distinct and important capability.
• OCR tokens can assist models in answering text-based questions.

Procedure

1. Select text-rich images Action: Choose images from OpenImages containing visible text. Why: Text-rich images are needed for text-reading questions. Note: See paper for details.
2. Collect questions Action: Have annotators write questions that require reading text in the image. Why: Human-written questions ensure natural, meaningful queries. Note: See paper for details.
3. Collect answers Action: Gather multiple answers per question from different annotators. Why: Multiple answers account for answer variation. Note: See paper for details.
4. Evaluate baseline models Action: Test existing VQA models with and without OCR on TextVQA. Why: Establishes baselines and demonstrates the need for text reading. Note: See paper for details.

Parameters to set

• images_count — Role: Number of text-rich images. How to set: Select diverse images with visible text. Default/range: 28K. Effect: More images increase diversity.
• questions_per_image — Role: Average questions per image. How to set: 1-2 questions focused on text. Default/range: ~1.6. Effect: More questions increase dataset utility.

Validation checks

• Questions should require reading text visible in the image to answer.
• Models without OCR should perform significantly worse than those with OCR.
• The dataset should cover diverse text appearances (signs, labels, screens).

Failure modes

• Some visible text may be too small or blurry to read.
• Questions may be answerable from context without reading the specific text.
• OCR errors may propagate to model answers.

Adaptation notes for VLM training

• TextVQA is a standard benchmark and training data source for VLMs with OCR capability.
• Include TextVQA in instruction tuning mixes for text reading ability.
• The dataset format has influenced DocVQA, ChartQA, and similar benchmarks.

Implementation notes

• Provide OCR tokens alongside images for model development.
• Use the VQA accuracy metric for evaluation.
• Track per-text-type performance (signs, labels, screens).

Evidence from the paper

• TextVQA provides 45K questions requiring reading and reasoning about text in images.
• Models without OCR capability struggle on TextVQA.
• The dataset reveals a significant gap in existing VQA models' text reading ability.
• TextVQA has become a standard benchmark for VLM text understanding.

Source paper

• Title: TextVQA: Towards Reasoning about Text in Images
• Year: 2019
• Venue: CVPR
• Paper ID: arxiv-1904.08920v2
• URL: http://arxiv.org/abs/1904.08920v2
• arXiv ID: 1904.08920v2