Faceid 6m Eval

faceid-6m-eval

FaceID-6M: A Large-Scale, Open-Source FaceID Customization Dataset — Wang et al. (2025) (arXiv:2503.07091, 2025)

What this evaluates

This evaluation protocol assesses the effectiveness of a large-scale dataset for training conditional diffusion models in FaceID customization. It probes the model's ability to preserve facial identity from a reference image while adhering to textual prompts and maintaining overall image quality.

Datasets

• COCO2017 — total 500; splits: test (500)
• Unsplash-50 — total 50; splits: test (50)

Metrics

• Face Sim (primary) — range: [0, 1]
  • Calculates the cosine similarity between the face embedding of the input reference image and the face embedding extracted from the generated image. Provides a direct estimate of facial identity preservation.
• CLIP-T — range: [0, 1]
  • Measures the cosine similarity between the text embedding of the input prompt and the image embedding of the generated image. Evaluates the model's ability to follow textual prompts.
• CLIP-I — range: [0, 1]
  • Measures the cosine similarity between the image embedding of the original reference face and the image embedding of the generated image. Assesses general image-level similarity after identity insertion.

Input / output format

Input: A reference face image and a natural language text prompt describing the desired image content.

Output: A single generated RGB image that incorporates the visual features of the reference face while matching the textual description.

Scoring recipe

def score(predictions, gold):
    input_face, prompt, orig_face = gold
    face_sim = cosine_similarity(face_encoder(input_face), face_encoder(predictions))
    clip_t = cosine_similarity(text_encoder(prompt), image_encoder(predictions))
    clip_i = cosine_similarity(image_encoder(orig_face), image_encoder(predictions))
    return {'Face Sim': face_sim, 'CLIP-T': clip_t, 'CLIP-I': clip_i}

Common pitfalls

• Using the full 5,000-image COCO2017 split instead of the manually curated 500 pairs, which introduces many non-face samples that skew FaceID fidelity metrics.
• Interpreting CLIP-I as a measure of facial identity; it actually measures general image-level similarity between the original reference and generated image, not necessarily preserving specific facial features.
• Comparing human evaluation scores (0–5 Likert scale) directly with automated cosine similarity scores (0–1 range) without normalization or clear separation of evaluation modalities.

Evidence (verbatim from paper)

For evaluation metrics, we use the following: (1) Face Sim, which calculates the FaceID cosine similarity between the input face and the face extracted from the generated image, providing a direct estimate of the difference between the generated and input faces. (2) CLIP-T [[22]], which evaluates the model’s ability to follow prompts; and (3) CLIP-I [[22]], which measures the CLIP image similarity between the original image and the image after FaceID insertion.

Citation

@misc{wang2025faceid6m,
  title={FaceID-6M: A Large-Scale, Open-Source FaceID Customization Dataset},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.07091}
}

• arXiv: 2503.07091