Multiid Bench Eval

multiid-bench-eval

WithAnyone: Towards Controllable and ID Consistent Image Generation — Hengyuan Xu et al. (2025) (arXiv:2510.14975, 2025)

What this evaluates

Evaluates image generation models on their ability to produce identity-consistent portraits while maintaining controllability over pose, expression, and lighting. It specifically probes the trade-off between accurate identity preservation and the generation of copy-paste artifacts from reference images.

Datasets

• MultiID-Bench — total ?; splits: single-person (-1), multi-person (-1); repo https://github.com/doby-xu/WithAnyone

Metrics

• face similarity (Sim(G)) (primary) — range: [0, 1]
  • Cosine similarity between face embeddings of the generated image and the ground-truth image, measuring identity preservation relative to the target.
• face similarity (Sim(R)) — range: [0, 1]
  • Cosine similarity between face embeddings of the generated image and the reference image, measuring how closely the output matches the provided ID reference.
• copy-paste (CP) — range: [0, 1]
  • A score quantifying the severity of copy-paste artifacts in the generated face, where lower values indicate fewer direct replicative artifacts.
• CLIP-I — range: [0, 1]
  • CLIP image similarity score measuring the visual quality and feature alignment of the generated image.
• CLIP-T — range: [0, 1]
  • CLIP text-image similarity score measuring alignment between the generated image and the input text prompt.
• Aes — range: other
  • Aesthetic quality score of the generated image, typically derived from a dedicated aesthetic predictor.

Input / output format

Input: Text prompt describing the scene, pose, expression, and lighting, plus one or more reference images containing the target identity/identities.

Output: Generated image(s) matching the prompt and reference identity/identities.

Scoring recipe

def compute_metrics(predictions, gold_prompts, reference_images, gt_images):
    gen_emb = get_face_embedding(predictions)
    ref_emb = get_face_embedding(reference_images)
    gt_emb = get_face_embedding(gt_images)
    
    sim_g = cosine_similarity(gen_emb, gt_emb)
    sim_r = cosine_similarity(gen_emb, ref_emb)
    cp_score = compute_copy_paste_artifact_score(predictions, reference_images)
    
    clip_i = compute_clip_image_similarity(predictions)
    clip_t = compute_clip_text_image_similarity(predictions, gold_prompts)
    aes = compute_aesthetic_score(predictions)
    
    return {
        'Sim(G)': sim_g,
        'Sim(R)': sim_r,
        'CP': cp_score,
        'CLIP-I': clip_i,
        'CLIP-T': clip_t,
        'Aes': aes
    }

Common pitfalls

• High face similarity scores can be artificially inflated by models directly copy-pasting reference facial features rather than synthesizing the identity, creating a misleading trade-off curve.
• VLM-based evaluators often fail to distinguish individual identities and instead score non-identity attributes like pose, expression, or background, leading to inaccurate identity fidelity measurements.
• The copy-paste metric may not perfectly align with human perception in all cases, though the paper notes a moderate positive correlation.

Evidence (verbatim from paper)

We observe a clear trade-off between face similarity and copy-paste artifacts. As shown in Fig.[5], most methods align closely with a regression curve, where higher face similarity generally coincides with stronger copy-paste. This indicates that many existing models boost measured similarity by directly replicating reference facial features rather than synthesizing the identity.

Citation

@misc{xu2025withanyone,
  title={WithAnyone: Towards Controllable and ID Consistent Image Generation},
  author={Hengyuan Xu et al. (2025)},
  year={2025},
  note={arXiv:2510.14975}
}

• arXiv: 2510.14975