Capsfusion Rethinking Image Text Data At Scale Arxiv 2310 20550v2

CapsFusion: Rethinking Image-Text Data at Scale

One-line decision

Use this skill when you want to fuse noisy web captions with synthetic model-generated captions using an LLM to get the best of both. Avoid it when you have only one caption source or cannot afford LLM-based fusion.

Skill metadata

• Skill type: caption-fusion
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Fuse raw web captions (which contain world knowledge and named entities) with synthetic model-generated captions (which are more visually descriptive) using an LLM, producing hybrid captions that combine the strengths of both.

Problem signature

• Modality: images with dual captions (web alt-text and synthetic) fused into a single improved caption.
• Data state: images with both original web captions and synthetic captions from a captioning model.
• Scale regime: 120M fused captions at scale.
• Model requirement: An LLM (e.g., ChatGPT) for fusing pairs of captions; a captioning model for generating synthetic captions.

Use when

• You have both noisy web captions and synthetic model-generated captions for the same images.
• You want to preserve world knowledge from web text while gaining visual accuracy from synthetic captions.
• You can use an LLM to merge caption pairs at scale.

Do not use when

• You have only one caption source.
• You cannot afford LLM inference for caption fusion.
• Web captions are entirely uninformative and contribute nothing.

Required inputs

• web_captions: Original noisy web alt-text for each image.
• synthetic_captions: Model-generated captions from a captioning model (e.g., BLIP2).
• fusion_llm: LLM used to fuse the two caption types into one.

Optional inputs

• fusion_prompt: Prompt template guiding the LLM to merge captions.
• quality_filter: Post-fusion filter to remove bad fusions.

Outputs

• fused_captions: Hybrid captions combining web knowledge and synthetic visual descriptions.
• fusion_log: Record of original captions and fused output for auditing.

Assumptions and prerequisites

• Web captions contain world knowledge (names, events, context) that synthetic captions lack.
• Synthetic captions provide better visual descriptions than noisy web text.
• An LLM can effectively merge complementary information from two caption types.

Procedure

1. Generate synthetic captions Action: Run a captioning model on all images to produce visually descriptive captions. Why: Creates the synthetic component for fusion. Note: See paper for details.
2. Design fusion prompt Action: Create an LLM prompt that instructs the model to merge web and synthetic captions, keeping the best of both. Why: The prompt controls fusion quality and what information is preserved. Note: See paper for details.
3. Fuse captions with LLM Action: Feed pairs of (web caption, synthetic caption) to the LLM with the fusion prompt. Why: LLM reasoning combines complementary information from both sources. Note: See paper for details.
4. Filter fused captions Action: Remove fused captions that are too generic, contradictory, or poorly formed. Why: Quality control ensures the fusion actually improved captions. Note: See paper for details.
5. Train VLM on fused data Action: Pre-train a VLM on the CapsFusion dataset and compare to web-only and synthetic-only baselines. Why: Validates that fusion outperforms using either caption type alone. Note: See paper for details.

Parameters to set

• fusion_prompt_template — Role: Instructions for the LLM to merge captions. How to set: Instruct to keep named entities from web, visual details from synthetic, and resolve conflicts. Default/range: See paper. Effect: Prompt quality directly affects fusion quality.
• synthetic_captioner — Role: Model generating the synthetic captions. How to set: Use BLIP2 or similar strong captioner. Default/range: BLIP2. Effect: Better captioner produces better fusion input.
• fusion_batch_size — Role: Number of caption pairs fused per LLM call. How to set: 1 pair per call for quality. Default/range: 1. Effect: Batching multiple pairs may degrade quality.

Validation checks

• Fused captions should contain world knowledge from web captions and visual details from synthetic ones.
• VLM trained on fused data should outperform both web-only and synthetic-only baselines.
• Caption length and informativeness should increase after fusion.

Failure modes

• The LLM may incorrectly resolve conflicts between web and synthetic captions.
• Fusion at scale is expensive due to LLM inference costs.
• The LLM may introduce hallucinations not present in either source caption.

Adaptation notes for VLM training

• The fusion approach can be applied to any pair of caption sources.
• Replace the LLM with Claude for potentially higher quality fusion.
• Extend to fuse more than two caption types (e.g., web + synthetic + OCR-derived).

Implementation notes

• Batch LLM calls efficiently to manage cost.
• Cache fusion results for reproducibility.
• Compare random samples of fused vs original captions to verify quality.

Evidence from the paper

• CapsFusion combines web captions (with world knowledge) and synthetic captions (with visual accuracy) using LLM-based fusion.
• Models trained on CapsFusion data outperform those trained on web-only or synthetic-only captions on 12 downstream tasks.
• CapsFusion produces 120M fused captions that preserve named entities and add visual descriptions.
• The fusion approach is complementary to other data curation methods.

Source paper

• Title: CapsFusion: Rethinking Image-Text Data at Scale
• Year: 2023
• Venue: CVPR
• Paper ID: arxiv-2310.20550v2
• URL: http://arxiv.org/abs/2310.20550v2
• arXiv ID: 2310.20550v2