Diffusion Instruction Tuning Eval

diffusion-instruction-tuning-eval

Diffusion Instruction Tuning — Chen Jin et al. (2025) (arXiv:2502.06814, 2025)

What this evaluates

This protocol evaluates the vision-language alignment and zero-shot generalization capabilities of fine-tuned VLMs across diverse multimodal tasks. It measures how effectively aligning VLM cross-attention with diffusion model attention maps improves performance on document understanding, reasoning, real-world visual comprehension, and hallucination detection benchmarks.

Datasets

• AI2D — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMMU — total ?; splits: val (-1)
• RealworldQA — total ?; splits: test (-1)
• SEED — total ?; splits: test (-1)
• HallucinationBench — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test (-1)
• OK-VQA — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• HatefulMemes — total ?; splits: test (-1)
• COCO — total ?; splits: val (-1)
• Flickr30K — total ?; splits: test (-1)
• WorldMedQA-V — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Exact-match or fuzzy-match accuracy against gold labels, computed per benchmark's default protocol. Reported as percentage or normalized score depending on the specific benchmark.

Input / output format

Input: Multimodal instances consisting of an image and a text prompt/question, formatted according to each VLM's specific tokenizer and instruction template.

Output: Text generation (answers, captions, or multiple-choice selections) produced autoregressively by the VLM.

Scoring recipe

def evaluate(prediction, gold, benchmark_name):
    if benchmark_name in ['VQAv2', 'OK-VQA', 'TextVQA', 'MMMU', 'ScienceQA']:
        return exact_match_or_fuzzy_match(prediction, gold)
    elif benchmark_name in ['COCO', 'Flickr30K']:
        return compute_cider_or_rouge(prediction, gold)
    elif benchmark_name in ['POPE', 'HallucinationBench']:
        return compute_precision_recall_f1(prediction, gold)
    else:
        return benchmark_default_metric(prediction, gold)

Common pitfalls

• Data overlap between fine-tuning datasets and benchmark test sets can artificially inflate scores; the paper uses a qualitative overlap scoring system to flag this risk.
• Benchmark metrics vary significantly across subsets (e.g., MMBench has four language subsets), requiring careful aggregation and separate reporting rather than a single averaged score.
• Cross-attention and self-attention VLM architectures respond differently to the alignment method, necessitating architecture-specific baselines to avoid misleading performance claims.

Evidence (verbatim from paper)

We evaluate these models on 20 standard VLM benchmarks, comparing them against 23 baseline models ([Section 7.2]). Results in [Figure 10] show up to 72% improvement in zero-shot accuracy, demonstrating Lavender’s effectiveness in vision-text alignment. All benchmarks are evaluated using their default metrics.

Citation

@misc{jin2025diffusioninstructiontuning,
  title={Diffusion Instruction Tuning},
  author={Chen Jin et al. (2025)},
  year={2025},
  note={arXiv:2502.06814}
}

• arXiv: 2502.06814