Livs T2i Alignment Eval

livs-t2i-alignment-eval

LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces — Mushkani et al. (2025) (arXiv:2503.01894, 2025)

What this evaluates

Evaluates how well text-to-image models align with pluralistic, intersectional community preferences for urban public space design. Probes whether multi-criteria preference optimization (DPO) improves alignment over a baseline, and how prompt origin and annotator demographics influence preference consistency and rating distributions.

Datasets

• LIVS — total 37710; splits: train (-1), test (2200)

Metrics

• preference_rate (primary) — range: percent
  • Percentage of pairwise comparisons where annotators preferred the DPO-aligned model over the baseline, alongside the percentage preferring the baseline and the percentage rating both as neutral. Calculated per criterion and overall.

Input / output format

Input: Text prompts describing urban public space designs (human-authored or AI-generated).

Output: Generated images from text-to-image models (baseline SDXL or DPO-finetuned SDXL).

Scoring recipe

# For each of the 2,200 held-out comparisons:
# 1. Generate images I_base and I_dpo from the same prompt.
# 2. Annotators rate I_base vs I_dpo on 3 criteria.
# 3. Record preference: 'base', 'dpo', or 'neutral' per criterion.
# 4. Collapse split ratings via majority vote if needed (for training, but eval reports per-criterion).
# 5. Compute:
#    dpo_rate = count(prefs == 'dpo') / total_comparisons
#    base_rate = count(prefs == 'base') / total_comparisons
#    neutral_rate = count(prefs == 'neutral') / total_comparisons
# Return rates per criterion and overall.

Common pitfalls

• Collapsing multi-criteria split ratings into a single binary label via majority vote obscures which specific dimensions drive preference.
• High neutral ratings (up to 50%) indicate ambiguity in community values rather than model failure, requiring careful interpretation of alignment success.
• Prompt origin (human vs GPT-4o) significantly affects neutrality rates, confounding model evaluation if prompts are not controlled.

Evidence (verbatim from paper)

Out of 2,200 new comparisons, annotators chose the DPO-aligned model in 700 (32%) instances and the baseline in 300 (14%), marking the remaining 1,100 (50%) as neutral (See Appendix [I]). Criteria with more training annotations (e.g., Comfort, Invitingness) showed stronger improvements under DPO, whereas Inclusivity and Diversity had higher neutral ratings

Citation

@misc{mushkani2025livs,
  title={LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces},
  author={Mushkani et al. (2025)},
  year={2025},
  note={arXiv:2503.01894}
}

• arXiv: 2503.01894