Tedigan Text To Face Eval

tedigan-text-to-face-eval

TediGAN: Text-Guided Diverse Face Image Generation and Manipulation — Xia et al. (2020) (arXiv:2012.03308, 2020)

What this evaluates

Evaluates a model's capability to synthesize high-resolution, diverse, and photorealistic face images conditioned on natural language prompts, and to perform text-guided editing of existing faces while preserving identity and irrelevant attributes.

Datasets

• Multi-Modal CelebA-HQ — total 30000; splits: test (-1); repo https://github.com/weihaox/TediGAN

Metrics

• FID (primary) — range: other (lower is better)
  • Fréchet Inception Distance: computes the Fréchet distance between feature distributions of real and generated images using Inception-v3 activations.
• LPIPS — range: other (lower is better)
  • Learned Perceptual Image Patch Similarity: measures perceptual difference between images using deep network features.
• Accuracy (Acc.) — range: [0, 100] percent
  • Percentage of user votes where the generated or manipulated image is judged to be most coherent with the given text description.
• Realism (Real.) — range: [0, 100] percent
  • Percentage of user votes where the image is judged to be more photo-realistic compared to baseline methods.

Input / output format

Input: Natural language text descriptions (for generation) or an input face image paired with a text description specifying desired edits (for manipulation).

Output: A 1024×1024 RGB face image.

Scoring recipe

def compute_metrics(real_imgs, gen_imgs, user_votes):
    fid = frechet_inception_distance(real_imgs, gen_imgs)
    lpips = learned_perceptual_image_patch_similarity(real_imgs, gen_imgs)
    # User study: 50 images, >20 raters each
    acc = sum(1 for v in user_votes if v == 'coherent') / len(user_votes) * 100
    real = sum(1 for v in user_votes if v == 'realistic') / len(user_votes) * 100
    return {'FID': fid, 'LPIPS': lpips, 'Accuracy': acc, 'Realism': real}

Common pitfalls

• FID and LPIPS are computed over large batches of randomly sampled text prompts, not a fixed validation set, making direct comparison sensitive to prompt distribution.
• Accuracy and Realism are derived from subjective user studies (50 images, >20 raters), introducing variance and lacking deterministic reproducibility.
• The Multi-Modal CelebA-HQ dataset is newly proposed; baselines must be retrained on it, so cross-paper comparisons require matching training protocols.

Evidence (verbatim from paper)

For evaluation, there are four important aspects: image quality, image diversity, accuracy, and realism. The quality of generated or manipulated images is evaluated through Fréchet Inception Distance (FID). The diversity is measured by the Learned Perceptual Image Patch Similarity (LPIPS). For image generation, the accuracy is evaluated by the similarity between the text and the corresponding generated image. For manipulation, the accuracy is evaluated by whether the modified visual attributes of the synthetic image are aligned with the given description and text-irrelevant contents are preserved. The accuracy and realism are evaluated through a user study, where the users are asked to judge which one is more photo-realistic, and more coherent with the given texts. We test accuracy and realism by randomly sampling 50 images with the same conditions and collect more than 20 surveys from different people with various backgrounds.

Citation

@misc{xia2020tedigan,
  title={TediGAN: Text-Guided Diverse Face Image Generation and Manipulation},
  author={Xia et al. (2020)},
  year={2020},
  note={arXiv:2012.03308}
}

• arXiv: 2012.03308