Gass T2i Diversity Eval

gass-t2i-diversity-eval

GASS: Geometry-Aware Spherical Sampling for Disentangled Diversity Enhancement in Text-to-Image Generation — Ye Zhu et al. (arXiv:2602.17200, 2026)

What this evaluates

Evaluates text-to-image generation models on their ability to produce diverse, high-quality, and semantically aligned images under fixed prompts. It specifically probes disentangled diversity by measuring prompt-dependent semantic variation versus prompt-independent background/style variation.

Datasets

• ImageNet-1K — total ?; splits: test (-1); HF imagenet-1k
• DrawBench — total ?; splits: test (-1); HF drawbench

Metrics

• Density — range: [0, 1]
  • Measures generative fidelity by computing the average nearest-neighbor distance between generated samples and reference images in feature space. Higher values indicate better coverage of the data distribution.
• Coverage — range: [0, 1]
  • Measures diversity by calculating the fraction of reference samples that have at least one generated neighbor within a threshold distance in feature space. Higher values indicate broader coverage of the data manifold.
• ClipScore — range: [0, 1]
  • Computes the cosine similarity between CLIP text and image embeddings to measure text-image alignment. Higher scores indicate stronger semantic consistency.
• VS (primary) — range: [0, 1]
  • A reference-free intrinsic diversity metric that measures the spread of generated image embeddings in CLIP space. Higher values indicate greater sample diversity without requiring ground-truth images.
• ImageReward — range: [-1, 1]
  • A reference-free perceptual quality metric trained on human preference data to score generated images based on aesthetic and alignment criteria. Higher scores indicate better perceived quality.
• SPP — range: [0, 1]
  • A proposed metric that quantifies the geometric spread of generated samples in CLIP space by decomposing embeddings into prompt-dependent and prompt-independent orthogonal components. Higher values indicate greater disentangled diversity.

Input / output format

Input: Text prompts (e.g., 'A photo of [class label]' for ImageNet; complex structural prompts for DrawBench)

Output: Generated images (50 per class for ImageNet, 10 per prompt for DrawBench)

Scoring recipe

def evaluate(generated_images, prompts, reference_images=None):
    # Reference-based metrics (ImageNet only)
    density = compute_density(generated_images, reference_images)
    coverage = compute_coverage(generated_images, reference_images)
    # Reference-free metrics (DrawBench & ImageNet)
    clip_score = compute_clip_score(prompts, generated_images)
    image_reward = compute_image_reward(prompts, generated_images)
    vs_diversity = compute_vs_diversity(generated_images)
    spp = compute_spp(generated_images)  # Geometric spread in CLIP space
    return {
        'density': density, 'coverage': coverage,
        'clip_score': clip_score, 'image_reward': image_reward,
        'vs': vs_diversity, 'spp': spp
    }

Common pitfalls

• VS and SPP are reference-free metrics, unlike Density and Coverage which require ground-truth reference images for computation.
• Diversity is explicitly disentangled into prompt-dependent (semantic) and prompt-independent (background/style) components; evaluating only one axis misses the method's core contribution.
• SPP is a novel proposed metric and is not available in standard evaluation libraries; it requires custom implementation based on CLIP embedding decomposition.

Evidence (verbatim from paper)

For ImageNet, we employ the classic Density and Coverage as indicators of fidelity and diversity, respectively. We complement these with ClipScore for alignment, and VS for intrinsic diversity. For DrawBench, due to the absence of reference images, we utilize reference-free metrics: ImageReward for perceptual quality, VS for diversity, and ClipScore for consistency. Additionally, we also report our proposed SPP to quantify the geometric spread of the generated samples.

Citation

@misc{zhu2026gass,
  title={GASS: Geometry-Aware Spherical Sampling for Disentangled Diversity Enhancement in Text-to-Image Generation},
  author={Ye Zhu et al.},
  year={2026},
  note={arXiv:2602.17200}
}

• arXiv: 2602.17200