adu2021 Problem Statement
Existing evaluation of multi-reference generation systems relies on global image quality metrics (FID, CLIP similarity) that conflate controllability with aesthetic quality. These metrics fail to answer diagnostic questions: "Which subject confused with which?" or "Did the model swap attributes or degrade quality?" This obscures fine-grained generation failures in multi-person scene synthesis, where visual attribute coherence is critical.
Dataset Construction
MultiBind comprises 508 instances drawn from real photographs with 1,527 human subjects. Each instance is annotated with:
- Per-subject ground-truth masks and bounding boxes: Precise segmentation isolating each subject region for controlled comparison
- Subject references via canonical transformation: Generatively normalize subject appearance to reduce inherent visual similarity confounds when computing confusion baselines
- Inpainted background references: Provide background context without subject interference
- Entity-indexed prompts: Average 474 words per instance, encoding detailed descriptions for each subject to enable fine-grained control
This construction enables isolating generation errors from inherent subject similarity biases.
Evaluation Protocol: Dimension-Wise Confusion Framework
The protocol decomposes multi-subject generation fidelity across independent dimensions, each evaluated by specialist models:
Face Identity Evaluation: InsightFace embeddings compute per-subject identity consistency and cross-subject confusion matrices, measuring whether face IDs remain bound to intended subjects.
Appearance Evaluation: Qwen3-VL embeddings capture clothing, texture, and visual style, detecting whether appearance attributes transfer between subjects.
Pose Evaluation: ViTPose extracts skeleton keypoints measuring body configuration consistency, revealing whether poses stay with intended subjects or drift.
Expression Evaluation: Qwen3-VL expression embeddings measure facial expression binding, detecting emotional attribute leakage across subjects.
Baseline-corrected similarity matrices isolate generation-induced changes from inherent subject similarity, enabling fair comparison across different subject pairs.
Diagnostic Failure Patterns
The framework identifies four distinct failure modes:
Drift: Subject visual quality degrades without confusing with other subjects. Indicates generation instability on that reference rather than cross-subject interference.
Swap: Attributes permute between subjects in systematic ways. Suggests the model assigns correct attributes but to wrong subject positions.
Dominance: One reference overwhelms multiple subjects, collapsing diversity. Indicates imbalanced attention or reference weighting.
Blending: Attributes average across subjects producing hybrid appearance. Reveals feature fusion failures in multi-subject attention mechanisms.
Experimental Findings
Testing six generators reveals sharp performance divergence. Closed-source models (Nano Banana Pro, GPT-Image-1.5) maintain 80%+ success rates across dimensions, while open-source alternatives show severe face confusion (>50% permutation error) and appearance swapping. Critically, these same models appear competitive on aggregate FID and CLIP metrics, demonstrating that holistic quality scores hide controllability failures.
When to Use
Use this benchmark when developing or evaluating multi-subject generation systems, particularly when fine-grained control over per-subject attributes is required. This is essential for applications like fashion synthesis, character animation, and multi-person scene generation where cross-subject contamination violates user intent.
Quality Validation Checklist
- Test all specialist models (face, appearance, pose, expression) on your generation outputs
- Compute baseline-corrected confusion matrices rather than raw similarity
- Categorize failures into drift, swap, dominance, or blending patterns
- Compare your model's aggregate metric scores against failure pattern rates to expose discrepancies
- Use pattern-specific failure analysis to guide architecture improvements