Re Laion Caption 19m Eval

re-laion-caption-19m-eval

Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M) — Merchant et al. (2025) (arXiv:2507.05300, 2025)

What this evaluates

This benchmark evaluates how well text-to-image models adhere to structured prompts by measuring the alignment between generated images and their corresponding captions. It probes the model's ability to preserve semantic details and follow prompt structure during fine-tuning.

Datasets

• Re-LAION-Caption 19M — total 19000000; splits: test (-1)

Metrics

• VQA LLaVA (primary) — range: [0, 1]
  • Average probability of the VQA model answering 'yes' to the query 'Is the figure showing: [caption]?' across all test set image-caption pairs.
• VQA InstructBLIP — range: [0, 1]
  • Average probability of the VQA model answering 'yes' to the query 'Is the figure showing: [caption]?' across all test set image-caption pairs.

Input / output format

Input: Generated image I_i and its corresponding caption c_i from the test set, paired with a standardized yes/no query template.

Output: Probability score for the answer 'yes' from the VQA model (LLaVA or InstructBLIP).

Scoring recipe

scores = []
for I, c in test_set:
    query = f"Is the figure showing: {c}?"
    prob_yes = vqa_model.predict(I, query, answer="yes")
    scores.append(prob_yes)
metric_value = sum(scores) / len(scores)

Common pitfalls

• Standard CLIP-based alignment metrics fail on this dataset due to the longer caption lengths exceeding the 77-token limit.
• Long fine-tuning durations can cause image color saturation and overfitting, confounding alignment results if not controlled via shorter training runs.
• Tokenizer artifacts (newlines, numbered lists) must be sanitized to prevent subword merging that degrades model input quality.

Evidence (verbatim from paper)

For text alignment evaluation we use two Visual Question Answering (VQA) metrics: LLaVA-based and InstructBLIP-based. For each generated image–caption pair $(I_{i},c_{i})$ in our test set, we form a standardized yes/no query: $Q_{i}$ = “Is the figure showing: $c_{i}$?” For each such query, the VQA models produces a probability for the answer “yes”. Thus, we report the average VQA score for the whole test set.

Citation

@misc{merchant2025structured,
  title={Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)},
  author={Merchant et al. (2025)},
  year={2025},
  note={arXiv:2507.05300}
}

• arXiv: 2507.05300