Iris Benchmark Eval

iris-benchmark-eval

Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs — Zhao et al. (2026) (arXiv:2603.00590, 2026)

What this evaluates

Probes fairness across understanding and generation tasks in Unified Multimodal Large Language Models (UMLLMs) by measuring Ideal Fairness, Real-world Fidelity, and Bias Inertia & Steerability across demographic attributes. It reveals systemic trade-offs, generation gaps, and personality splits that single-task or single-metric evaluations miss.

Datasets

• IRIS Benchmark — total ?; splits: test (-1)

Metrics

• IRIS-Score (primary) — range: other
  • Aggregated score across three dimensions (Ideal Fairness, Real-world Fidelity, Bias Inertia & Steerability) for both understanding and generation tasks. Higher values indicate better fairness performance.
• Ideal Fairness (IFS) — range: other
  • Measures alignment with normative fairness principles across demographic attributes (age, gender, skin tone) in model outputs.
• Real-world Fidelity (RFS) — range: other
  • Assesses how accurately and fairly models reflect real-world demographic distributions and contexts.
• Bias Inertia & Steerability (BIS) — range: other
  • Evaluates the model's willingness and ability to adjust outputs when prompted to reduce bias or change demographic representation.

Input / output format

Input: Multimodal prompts (text and/or images) requiring the model to generate or interpret content involving demographic attributes (age, gender, skin tone) for fairness assessment.

Output: Generated text and/or images, which are subsequently analyzed by the ARES classifier and scored across the three fairness dimensions.

Scoring recipe

def compute_iris_scores(predictions, gold_demographics):
    # 1. Use ARES classifier to detect age, gender, skin tone in outputs
    detected = ares_classifier.predict(predictions)
    # 2. Calculate dimension scores based on fairness criteria
    ifs = calculate_ideal_fairness(detected, gold_demographics)
    rfs = calculate_real_world_fidelity(detected, gold_demographics)
    bis = calculate_bias_inertia_steerability(detected, gold_demographics)
    # 3. Aggregate into overall IRIS-Score
    iris_score = aggregate_scores(ifs, rfs, bis)
    return {'IFS': ifs, 'RFS': rfs, 'BIS': bis, 'IRIS-Score': iris_score}

Common pitfalls

• Assuming a single optimal model exists; the benchmark explicitly demonstrates inherent trade-offs across fairness dimensions.
• Evaluating understanding and generation tasks in isolation; the synchronous dual-task design is required to detect phenomena like the 'generation gap' and 'personality splits'.
• Treating Bias Inertia & Steerability (BIS) as a monolithic metric; validation shows it comprises distinct 'willingness' and 'ability' components.

Evidence (verbatim from paper)

All scores are scaled such that higher values indicate better performance (↑). For each metric, the best performance (highest score) is marked with † and the worst (lowest) with ‡. The left panel details the fairness scores across Understanding (Und) and Generation (Gen) tasks. The right panel displays the overall IRIS-Score and the diagnosed model personality profiles.

Citation

@misc{zhao2026irisbenchmark,
  title={Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.00590}
}

• arXiv: 2603.00590