Mosaig Multicultural T2i Eval

mosaig-multicultural-t2i-eval

Multi-Agent Multimodal Models for Multicultural Text to Image Generation — Bhalerao et al. (2025) (arXiv:2502.15972, 2025)

What this evaluates

Evaluates text-to-image generation models on their ability to produce culturally diverse, demographically accurate images with correct landmark representation. It probes alignment, visual quality, aesthetic appeal, cross-cultural knowledge, and demographic fairness across multiple languages and intersectional attributes.

Datasets

• MosAIG Dataset — total 9000; splits: test (9000); repo https://github.com/OanaIgnat/MosAIG

Metrics

• CLIPScore (primary) — range: [-1, 1]
  • Cosine similarity between the semantic embeddings of the generated image and its associated text caption. Higher values indicate stronger alignment.
• Inception Score (IS) — range: other
  • Measures image fidelity and diversity using an Inception v3 classifier. Computed over a batch of generated images.
• Aesthetic Score — range: [1, 10]
  • Rating of visual appeal (clarity, sharpness, color vibrancy, composition) predicted by a SigLIP-based model.
• Fairness — range: [0, 2]
  • Absolute difference in CLIPScore between the original caption-image pair and a demographic-modified pair: ΔS = |S(c,I) - S(c',I')|. Lower values indicate less bias.
• Knowledge — range: [0, 2]
  • Absolute difference in CLIPScore when a historical landmark in the caption is swapped: ΔS = |S(c,I) - S(c',I)|. Higher values indicate better landmark recognition.

Input / output format

Input: Text caption specifying demographic attributes (age, gender, nationality) and cultural landmarks, optionally in English, Hindi, Spanish, Vietnamese, or other languages.

Output: Generated image corresponding to the input caption.

Scoring recipe

def evaluate(predictions, gold_captions, clip_model, aesthetic_model):
    scores = {}
    for pred_img, gold_c in zip(predictions, gold_captions):
        clip_val = clip_model.similarity(gold_c, pred_img)
        scores['CLIPScore'] = clip_val
        scores['Aesthetic'] = aesthetic_model.predict(pred_img)
        c_fair, c_know = modify_demographics(gold_c), modify_landmark(gold_c)
        img_fair, img_know = generate(c_fair), generate(c_know)
        scores['Fairness'] = abs(clip_val - clip_model.similarity(c_fair, img_fair))
        scores['Knowledge'] = abs(clip_val - clip_model.similarity(c_know, img_know))
    scores['Inception Score'] = inception_score(predictions)
    return scores

Common pitfalls

• Fairness and Knowledge both use absolute CLIPScore differences but have opposite optimal directions (lower is better for Fairness, higher for Knowledge).
• Inception Score and Aesthetic Score are computed over batches or require external models, making per-instance scoring non-trivial.
• Human evaluation relies on weighted Cohen’s Kappa with only moderate agreement (0.5–0.6), indicating high annotator subjectivity for visual metrics.

Evidence (verbatim from paper)

CLIPScore measures text-to-image alignment by computing the cosine similarity between the semantic embeddings of the image and its associated text, providing an effective assessment of how well the generated image reflects the intended description. CLIPScore ranges from -1 to +1, where higher values indicate a stronger semantic alignment between the generated image and its corresponding text.

Citation

@misc{bhalerao2025mosaig,
  title={Multi-Agent Multimodal Models for Multicultural Text to Image Generation},
  author={Bhalerao et al. (2025)},
  year={2025},
  note={arXiv:2502.15972}
}

• arXiv: 2502.15972