Omnigen2 Eval

omnigen2-eval

OmniGen2: Exploration to Advanced Multimodal Generation — Chenyuan Wu et al. (2025) (arXiv:2506.18871, 2025)

What this evaluates

Evaluates a unified multimodal model's capabilities across visual understanding, text-to-image generation, instruction-based image editing, and in-context generation. It probes compositional prompt following, long-prompt adherence, edit accuracy versus preservation, and subject consistency across single, multiple, and scene contexts.

Datasets

• MMBench — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)
• GenEval — total ?; splits: test (-1)
• DPG-Bench — total ?; splits: test (-1)
• Emu-Edit — total ?; splits: test (-1)
• GEdit-Bench-EN — total ?; splits: test (-1)
• ImgEdit-Bench — total ?; splits: test (-1)
• OmniContext — total ?; splits: test (-1)

Metrics

• GenEval Overall (primary) — range: [0, 1]
  • Average score across six compositional categories: single object, two objects, counting, colors, position, and color attribution. Scores range from 0 to 1 per category.
• DPG-Bench Overall (primary) — range: [0, 100]
  • Average score across five dimensions: global, entity, attribute, relation, and other. Measures long prompt following capability.
• Emu-Edit CLIP-Out — range: [0, 1]
  • CLIP feature similarity between the generated image and the target caption. Higher values indicate better alignment with the edit instruction.
• GEdit-Bench-EN SC — range: [0, 10]
  • Semantic Consistency score evaluating instruction following. Assessed via human or automated scoring on a scale up to 10.
• ImgEdit-Bench Overall — range: [0, 10]
  • Average score across nine editing task types (Add, Adjust, Extract, Replace, Remove, Background, Style, Hybrid, Action).
• OmniContext Overall — range: [0, 10]
  • Average of Prompt Following (PF) and Subject Consistency (SC) scores across task splits (Single, Multiple, Scene). Higher is better.

Input / output format

Input: Text prompts, reference images (for editing and in-context generation), and optional editing instructions or context images.

Output: Generated images corresponding to the input prompts and instructions.

Scoring recipe

def compute_metrics(predictions, gold):
    scores = {}
    # GenEval: average category scores
    scores['GenEval_Overall'] = np.mean([cat_score for cat_score in predictions['gen_eval_categories']])
    # DPG-Bench: average dimension scores
    scores['DPG_Overall'] = np.mean([dim_score for dim_score in predictions['dpg_dimensions']])
    # Emu-Edit: CLIP/DINO similarity
    scores['EmuCLIP_Out'] = clip_similarity(predictions['image'], gold['target_caption'])
    scores['EmuCLIP_I'] = clip_similarity(predictions['image'], gold['source_image'])
    scores['EmuDINO'] = dino_similarity(predictions['image'], gold['source_image'])
    # GEdit/ImgEdit/OmniContext: average task/subtask scores
    scores['GEdit_SC'] = predictions['gedit_sc']
    scores['ImgEdit_Overall'] = np.mean(predictions['imgedit_tasks'])
    scores['OmniContext_Overall'] = np.mean(predictions['omni_context_pf_sc'])
    return scores

Common pitfalls

• Different benchmarks use different scoring scales (0-1, 0-100, 0-10), which can cause confusion when comparing overall model performance across tasks.
• GenEval and DPG-Bench report both sub-category scores and an overall average; readers must specify which is being reported to avoid misinterpretation.
• The OmniContext benchmark is newly proposed and uses task-specific splits (Single, Multiple, Scene) that are not standardized in external repositories, making direct reproduction difficult without the paper's code.
• Image editing metrics like CLIP-I and DINO measure preservation of unedited regions, while CLIP-Out measures alignment with the edit instruction; optimizing one often trades off against the other.

Evidence (verbatim from paper)

We assess OmniGen2’s T2I generation capabilities on two standard benchmarks: GenEval, which evaluates compositional understanding, and DPG-Bench, which measures long prompt following. Our model demonstrates highly competitive performance, particularly when considering its resource efficiency. ... On Emu-Edit, our model achieves the highest CLIP-Out score (0.309), indicating it most effectively applies the requested edits among all compared models. Concurrently, it secures the second-best scores for CLIP-I (0.876) and DINO (0.822), which measure the preservation of unedited regions.

Citation

@misc{wu2025omnigen2,
  title={OmniGen2: Exploration to Advanced Multimodal Generation},
  author={Chenyuan Wu et al. (2025)},
  year={2025},
  note={arXiv:2506.18871}
}

• arXiv: 2506.18871