Emu35 T2i X2i Eval

emu35-t2i-x2i-eval

Emu3.5: Native Multimodal Models are World Learners — Yufeng Cui et al. (2025) (arXiv:2510.26583, 2025)

What this evaluates

Evaluates a multimodal model's capability to generate images from text prompts and edit existing images based on natural language instructions. It probes semantic alignment, fine-grained text rendering accuracy, and instruction-following fidelity across diverse visual tasks.

Datasets

• GenEval — total ?; splits: test (-1)
• DPG-bench — total ?; splits: test (-1)
• OneIG-Bench — total ?; splits: test (-1)
• TIIF-Bench mini — total ?; splits: test (-1)
• LeX-Bench — total 1310; splits: test (1310)
• CVTG-2K — total 2000; splits: test (2000)
• LongText-Bench — total ?; splits: test (-1)
• ImgEdit — total 737; splits: test (737)
• GEdit-Bench — total 606; splits: test (606)
• OmniContext — total ?; splits: test (-1)
• ICE-Bench — total ?; splits: test (-1)

Metrics

• Word Accuracy (primary) — range: percent
  • Exact match rate of rendered text characters in the generated image compared to the prompt. Higher is better.
• Normalized Edit Distance (NED) — range: [0, 1]
  • 1 - (Edit Distance / max(len(pred), len(gold))). Measures character-level similarity; higher is better.
• G_O — range: [0, 5]
  • Overall score computed as the mean of GPT-4o/GPT-4.1 judge scores across all samples in the benchmark. Not the mean of sub-metrics.
• Overall (OneIG-Bench) — range: [0, 1]
  • Average of five dimensions: Alignment, Text, Reasoning, Style, and Diversity.

Input / output format

Input: Text prompt (for T2I) or image(s) + natural language instruction (for X2I/editing).

Output: Generated image (T2I) or edited image (X2I). Evaluated at ~1024x1024 resolution for X2I, up to 2048px for T2I.

Scoring recipe

def compute_metrics(predictions, golds, benchmark):
    if benchmark == 'CVTG-2K':
        word_acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
        ned = [1 - edit_distance(p, g) / max(len(p), len(g)) for p, g in zip(predictions, golds)]
        return {'Word Accuracy': word_acc, 'NED': mean(ned)}
    elif benchmark in ['ImgEdit', 'GEdit-Bench']:
        # LLM-as-judge scoring (GPT-4.1 or GPT-4o)
        scores = [llm_judge_score(p, g, benchmark) for p, g in zip(predictions, golds)]
        return {'G_O': mean(scores)}
    elif benchmark == 'OneIG-Bench':
        dims = ['Alignment', 'Text', 'Reasoning', 'Style', 'Diversity']
        dim_scores = [llm_judge_score(p, g, dim) for dim in dims]
        return {'Overall': mean(dim_scores)}
    return {'score': -1}

Common pitfalls

• G_O in GEdit-Bench is the mean score across all samples, not the arithmetic mean of G_SC and G_PQ.
• X2I evaluations must use ~1024x1024 resolution; T2I supports up to 2048px. Mismatched resolutions invalidate comparisons.
• Different benchmarks use different LLM judges (GPT-4o vs GPT-4.1); mixing them or assuming identical scoring rubrics leads to incorrect rankings.

Evidence (verbatim from paper)

For evaluation, two specific metrics are employed: Word Accuracy and Normalized Edit Distance (NED). As illustrated in the table, Emu3.5 outperforms state-of-the-art T2I models by a large margin, further demonstrating its strong capability in English text rendering.

Citation

@misc{cui2025emu35,
  title={Emu3.5: Native Multimodal Models are World Learners},
  author={Yufeng Cui et al. (2025)},
  year={2025},
  note={arXiv:2510.26583}
}

• arXiv: 2510.26583