Rovi Instance Grounding Eval

rovi-instance-grounding-eval

ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation — Peng et al. (2025) (arXiv:2508.01008, 2025)

What this evaluates

Evaluates the ability of text-to-image models to accurately render specific objects at requested bounding box locations while maintaining prompt fidelity, attribute correctness, and overall aesthetic quality.

Datasets

• ROVI validation set — total ?; splits: val (-1); repo https://github.com/CihangPeng/ROVI

Metrics

• Gen Inst. (primary) — range: [0, 1]
  • Per-box cross-checking pass rate using VLM-based evaluation. Each expected instance is cropped from its specified bounding box on the generated image, and a VLM (Qwen2VL) computes the likelihood that the cropped window matches its assigned label. The metric is the fraction of boxes that pass this check.
• FID — range: [0, ∞)
  • Fréchet Inception Distance measuring the distance between feature distributions of real and generated images. Lower values indicate higher quality.
• Aes. — range: [0, 10]
  • Aesthetic score predicting human preference for image quality. Higher values indicate better aesthetics.
• CLIP Sim. — range: [0, 1]
  • CLIP-based image-text similarity measuring alignment between the generated image and the input prompt. Higher values indicate better prompt fidelity.

Input / output format

Input: Text prompt with associated bounding box coordinates and instance labels (box-label pairs).

Output: Generated image.

Scoring recipe

def score_gen_inst(predictions, gold):
    passes = 0
    total = 0
    for img, (_, box_labels) in zip(predictions, gold):
        for box, label in box_labels:
            crop = img.crop(box)
            if vlm_check(crop, label):  # Qwen2VL returns match
                passes += 1
            total += 1
    return passes / total if total > 0 else 0.0

Common pitfalls

• Using standard detection datasets like Open Images for evaluation yields artificially low scores due to limited category vocabularies (e.g., 600 categories), inconsistent captions, and poor aesthetic quality for generation tasks.
• Image-level metrics like FID and CLIP similarity often disadvantage fine-tuned models; improvements in grounding and prompt quality are required to offset this baseline penalty.

Evidence (verbatim from paper)

Gen Inst.: per-box cross-checking pass rate using VLM-based evaluation. Our G.: GLIGEN trained on ROVI. SD v1.4: Stable Diffusion baseline*[[Rombach et al.(2022a)Rombach, Blattmann, Lorenz, Esser, and Ommer]]*.

Citation

@misc{peng2025rovi,
  title={ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2508.01008}
}

• arXiv: 2508.01008