X Omni Eval

x-omni-eval

X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again — Geng et al. (2025) (arXiv:2507.22058, 2025)

What this evaluates

Evaluates the text rendering, text-to-image generation, and image understanding capabilities of a discrete autoregressive image generation model trained with reinforcement learning. It probes the model's ability to follow complex instructions, render long texts accurately, and generate high-fidelity images without relying on classifier-free guidance.

Datasets

• OneIG-Bench — total ?; splits: test (-1)
• LongText-Bench — total 160; splits: test (160)
• DPG-Bench — total ?; splits: test (-1)
• GenEval — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• SEEDBench-Img — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)

Metrics

• DPG-Bench Overall (primary) — range: [0, 100]
  • Composite score averaging category-level scores (Global, Entity, Attribute, Relation, Other) evaluated by a vision-language model comparing generated images to prompts.
• GenEval Overall — range: [0, 1]
  • Composite score averaging category-level scores (Single, Two, Counting, Colors, Position, Color Attr.) evaluating precise attribute and spatial reasoning in generated images.
• OneIG-Bench Text Rendering — range: [0, 1]
  • Composite score derived from Edit Distance, Completion Rate, and Word Accuracy to assess text rendering proficiency in English and Chinese.
• LongText-Bench — range: [0, 1]
  • Score evaluating the capacity to precisely render long Chinese and English texts across 8 scenarios using 160 prompts.
• POPE — range: [0, 1]
  • Accuracy/F1 score for object hallucination detection in image understanding.
• GQA — range: [0, 1]
  • Accuracy score for visual question answering focusing on compositional reasoning.
• MMBench — range: [0, 1]
  • Accuracy score for multimodal benchmark evaluating general multimodal understanding.
• SEEDBench-Img — range: [0, 1]
  • Accuracy score for image understanding across multiple tasks.
• DocVQA — range: [0, 1]
  • Exact match accuracy for document visual question answering.
• OCRBench — range: other
  • OCR score measuring optical character recognition capability on images.

Input / output format

Input: Text prompts for generation benchmarks; image-text pairs for understanding benchmarks.

Output: Generated images for generation tasks; text answers for understanding tasks.

Scoring recipe

def evaluate_generation(pred_images, prompts, benchmark):
    scores = []
    for img, prompt in zip(pred_images, prompts):
        if benchmark == 'GenEval':
            prompt = rewrite_prompt(prompt)
        scores.append(vlm_score(img, prompt))
    return mean(scores)

def evaluate_understanding(pred_answers, gold_answers, benchmark):
    if benchmark in ['POPE', 'GQA', 'MMBench', 'SEEDBench-Img', 'DocVQA']:
        return exact_match_or_f1(pred_answers, gold_answers)
    elif benchmark == 'OCRBench':
        return ocr_score(pred_answers, gold_answers)
    return 0.0

Common pitfalls

• Prompt rewriting is required for GenEval evaluation; skipping it invalidates results.
• GPT-4o API rejects certain prompts, so reported results exclude these cases.
• Classifier-free guidance (CFG) significantly impacts autoregressive generation quality; results without CFG should be reported separately to avoid misleading comparisons.

Evidence (verbatim from paper)

We evaluate text-to-image generation on two widely recognized benchmarks: DPG-Bench*[hu2024dpgbench]* and GenEval*[ghosh2023geneval]*. Detailed results are shown in Table[2] and Table[3] respectively. Note that prompt rewriting is employed for GenEval evaluation.

Citation

@misc{geng2025xomni,
  title={X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again},
  author={Geng et al. (2025)},
  year={2025},
  note={arXiv:2507.22058}
}

• arXiv: 2507.22058