Multiref Bench Eval

multiref-bench-eval

MultiRef: Controllable Image Generation with Multiple Visual References — Chen et al. (2025) (arXiv:2508.06905, 2025)

What this evaluates

Evaluates the ability of image generation models to simultaneously align and incorporate multiple visual reference conditions (e.g., bounding boxes, depth maps, masks, sketches) alongside text instructions. It probes complex multi-source creative synthesis, testing both global image quality and fine-grained reference fidelity across different input formats and processing orders.

Datasets

• MULTIREF-BENCH — total 1990; splits: test (1990)

Metrics

• Overall Assessment (IQ, IF, SF) (primary) — range: [0, 1]
  • Composite score comprising Image Quality (IQ), Instruction Following (IF), and Source Fidelity (SF). Evaluated via LLM/human judgment on a normalized 0–1 scale where higher is better.
• FID — range: [0, inf)
  • Fréchet Inception Distance measuring distributional similarity between generated and reference image sets. Lower values indicate better quality.
• Aesthetic — range: [0, 1]
  • Aesthetic score evaluating visual appeal and artistic quality. Higher values indicate better aesthetics.
• Reference Fidelity (AVG, BBox, Semantic Map, Mask, Depth Map, Canny Edge, Sketch, Caption, Pose, Subject, Art Style) — range: [0, 1] or error units
  • Fine-grained fidelity metrics measuring adherence to specific visual conditions. BBox, Semantic Map, Mask, Caption, Pose, Subject, and Art Style are higher-is-better (↑). Depth Map, Canny Edge, and Sketch are lower-is-better (↓), typically computed as MSE or error rates. AVG is the mean of all reference fidelity scores.

Input / output format

Input: Multiple reference images (e.g., BBox, depth, mask, canny, sketch, pose, subject, art style) combined with a text instruction/caption. For compositional setups, references are provided sequentially in multi-turn dialogues with one reference per turn.

Output: A single generated image.

Scoring recipe

# IQ, IF, SF, and Reference Fidelity are LLM/human judged (0-1 scale)
# FID, Depth Map, Canny Edge, Sketch are lower-is-better (↓)
# Aesthetic, BBox, Semantic Map, Mask, Caption, Pose, Subject, Art Style are higher-is-better (↑)
metrics = {}
metrics['FID'] = compute_fid(generated, ref_images)
metrics['Aesthetic'] = compute_aesthetic(generated)
metrics['IQ'] = judge_score(generated, prompt, 'image_quality')
metrics['IF'] = judge_score(generated, prompt, 'instruction_following')
metrics['SF'] = judge_score(generated, prompt, 'source_fidelity')
for cond in ['BBox', 'Semantic Map', 'Mask', 'Caption', 'Pose', 'Subject', 'Art Style']:
    metrics[cond] = fidelity_score(generated, ref_condition, cond)
for cond in ['Depth Map', 'Canny Edge', 'Sketch']:
    metrics[cond] = error_score(generated, ref_condition, cond)
return metrics

Common pitfalls

• Input order drastically changes specific conditional fidelities (e.g., depth/sketch error) without significantly affecting global FID or overall image quality.
• Removing captions has mixed effects: it improves sketch and semantic map fidelity but degrades depth fidelity and aesthetic quality across models.
• Compositional LLM+Diffusion pipelines score higher on image quality but consistently underperform on instruction following and source fidelity compared to unified end-to-end models.

Evidence (verbatim from paper)

As shown in Figure 4, LLM+SD combinations achieve the highest image quality scores, with Claude + SD3.5 reaching 0.774, occasionally surpassing ground truth. However, all compositional frameworks consistently underperform in instruction following and source fidelity. While ground truth achieves 0.767 and 0.706 for IF and SF respectively, Claude + SD3.5 only reaches 0.589 and 0.462, indicating that a separated perceptron-generator architecture fundamentally compromises complex visual instruction execution.

Citation

@misc{chen2025multiref,
  title={MultiRef: Controllable Image Generation with Multiple Visual References},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.06905}
}

• arXiv: 2508.06905