Dreambench Customization Eval

dreambench-customization-eval

MUSAR: Exploring Multi-Subject Customization from Single-Subject Dataset via Attention Routing — Guo et al. (2025) (arXiv:2505.02823, 2025)

What this evaluates

Evaluates a text-to-image model's ability to customize generated images with specific subjects from reference images, both individually and in combination, while maintaining alignment with text prompts.

Datasets

• DreamBench — total 750; splits: test (750)

Metrics

• CLIP-I (primary) — range: [0, 1]
  • Cosine similarity between CLIP image features of the generated image and the reference image(s). For multi-subject tasks, averaged across all reference images.
• DINO — range: [0, 1]
  • Cosine similarity between DINO image features of the generated image and the reference image(s). For multi-subject tasks, averaged across all reference images.
• CLIP-T — range: [0, 1]
  • Cosine similarity between CLIP image features of the generated image and CLIP text features of the prompt.

Input / output format

Input: Text prompt and one or more reference images specifying the subjects to customize.

Output: A single generated image matching the prompt and incorporating the reference subjects.

Scoring recipe

def compute_metrics(generated_imgs, ref_imgs, prompts, clip_enc, dino_enc):
    clip_i, dino, clip_t = [], [], []
    for gen, refs, prompt in zip(generated_imgs, ref_imgs, prompts):
        g_feat = clip_enc(gen)
        r_feats = [clip_enc(r) for r in refs]
        clip_i.append(mean([cos_sim(g_feat, rf) for rf in r_feats]))
        g_dino = dino_enc(gen)
        r_dino = [dino_enc(r) for r in refs]
        dino.append(mean([cos_sim(g_dino, rd) for rd in r_dino]))
        p_feat = clip_enc(prompt)
        clip_t.append(cos_sim(g_feat, p_feat))
    return mean(clip_i), mean(dino), mean(clip_t)

Common pitfalls

• Results vary significantly with random seeds; the protocol requires averaging over 4 seeds per sample.
• CLIP-T measures text-image alignment, not subject fidelity or attribute preservation.
• Multi-subject metrics require averaging similarity across all reference images, not just the first one.

Evidence (verbatim from paper)

We evaluate our method’s performance across both single-subject and multi-subject customization tasks. For single-subject evaluation, we employ the complete set of 750 test samples from the DreamBench dataset. For multi-subject scenarios, we construct test cases by pairing subjects from DreamBench to create 60 unique pairs, along with 20 composed triplets, resulting in a comprehensive set of 80 multi-subject test samples. Following previous works, we measured the model’s quantitative performance through image and text fidelity. For image fidelity, we used the CLIP and DINO models to calculate the cosine similarity between the generated images and the reference images, referred to as CLIP-I and DINO, respectively. To evaluate multi-subject generation, we extended CLIP-I and DINO by computing the average similarity between each generated image and all corresponding reference images. For text fidelity, we used the CLIP model to calculate the cosine similarity between the generated images and the text prompts, which is known as CLIP-T. To ensure statistical reliability, each test sample was generated with four different random seeds.

Citation

@misc{guo2025musar,
  title={MUSAR: Exploring Multi-Subject Customization from Single-Subject Dataset via Attention Routing},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2505.02823}
}

• arXiv: 2505.02823