Dreambench Subject Control Eval

dreambench-subject-control-eval

Less-to-More Generalization: Unlocking More Controllability by In-Context Generation — Shaojin Wu et al. (2025) (arXiv:2504.02160, 2025)

What this evaluates

Evaluates a diffusion model's ability to generate images that preserve subject identity from reference images while adhering to text prompts, covering both single-subject and multi-subject scenarios.

Datasets

• DreamBench — total ?; splits: test (-1)

Metrics

• DINO (primary) — range: [0, 1]
  • Cosine similarity between DINO embeddings of the generated image and the reference image. Higher values indicate better subject preservation.
• CLIP-I — range: [0, 1]
  • Cosine similarity between CLIP image embeddings of the generated image and the reference image. Used as a secondary measure of subject similarity.
• CLIP-T — range: [0, 1]
  • Cosine similarity between CLIP text embeddings of the prompt and CLIP image embeddings of the generated image. Measures text fidelity/adherence.

Input / output format

Input: One or more reference images (subject(s)) and a text prompt describing the desired scene.

Output: A single generated image per prompt.

Scoring recipe

def score(generated_img, ref_img, prompt):
    gen_emb = get_embeddings(generated_img)
    ref_emb = get_embeddings(ref_img)
    txt_emb = get_text_embeddings(prompt)
    dino = cosine_sim(gen_emb['dino'], ref_emb['dino'])
    clip_i = cosine_sim(gen_emb['clip_img'], ref_emb['clip_img'])
    clip_t = cosine_sim(gen_emb['clip_img'], txt_emb['clip_txt'])
    return {'DINO': dino, 'CLIP-I': clip_i, 'CLIP-T': clip_t}

Common pitfalls

• CLIP-I measures image-to-image similarity (subject fidelity), while CLIP-T measures text-to-image similarity (prompt adherence); confusing them leads to incorrect capability assessment.
• DreamBench requires generating 6 images per prompt and averaging scores; reporting single samples or different counts breaks comparability with the benchmark.
• Multi-subject evaluation uses 30 specific subject combinations from DreamBench; using arbitrary subjects invalidates the benchmark protocol.

Evidence (verbatim from paper)

Following previous works, we use standard automatic metrics to evaluate both subject similarity and text fidelity. Specifically, we employ cosine similarity measures between generated images and reference images within CLIP*[[27]]* and DINO*[[22]]* spaces, referred to as CLIP-I and DINO scores, respectively, to assess subject similarity. Additionally, we calculate the cosine similarity between the prompt and the image CLIP embeddings (CLIP-T) to evaluate text fidelity.

Citation

@misc{wu2025lesstomore,
  title={Less-to-More Generalization: Unlocking More Controllability by In-Context Generation},
  author={Shaojin Wu et al. (2025)},
  year={2025},
  note={arXiv:2504.02160}
}

• arXiv: 2504.02160