Deepgen 1.0 Eval

deepgen-1.0-eval

DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing — Wang et al. (2026) (arXiv:2602.12205, 2026)

What this evaluates

Evaluates a unified multimodal model's capabilities in text-to-image generation, image editing, and world-knowledge reasoning. It probes semantic alignment, long-horizon instruction following, fine-grained attribute binding, and precise text rendering across diverse scenarios.

Datasets

• GenEval — total ?; splits: test (-1)
• DPG-Bench — total ?; splits: test (-1)
• UniGenBench — total ?; splits: test (-1)
• WISE — total 1000; splits: test (1000)
• T2I-CoREBench — total ?; splits: test (-1)
• ImgEdit — total ?; splits: test (-1)
• GEdit-EN — total ?; splits: test (-1)
• UniREditBench — total 2700; splits: test (2700)
• RISE — total 327; splits: test (327)
• CVTG-2K — total 2000; splits: test (2000)

Metrics

• GenEval (primary) — range: [0, 1]
  • Automated semantic alignment score computed using the official GenEval LLM-based evaluation script. Measures how well generated images match the semantic content of text prompts.
• DPG-Bench — range: other
  • Instruction-following score based on LLM-as-a-judge evaluations of long-prompt adherence and image quality.
• UniGenBench — range: other
  • Comprehensive fine-grained generation score aggregating performance across ten categories including attribute binding, style control, and text rendering.
• WISE — range: [0, 1]
  • World-knowledge reasoning generation score evaluating cultural, scientific, and spatial-temporal understanding.
• T2I-CoREBench — range: other
  • Reasoning generation score across eight structured categories (Logical, Behavioral, Hypothetical, etc.).
• GEdit-EN — range: other
  • Image editing score measuring instruction following, editing consistency, and output quality.
• RISE — range: other
  • Reasoning-based editing score across temporal, causal, spatial, and logical dimensions.
• UniREditBench — range: other
  • World-knowledge-grounded editing score covering real- and game-world scenarios.
• CVTG-2K Word Accuracy — range: [0, 1]
  • Character-level correctness and legibility score for English text generation in diverse scenarios.

Input / output format

Input: Text prompt for generation tasks; text prompt plus reference image for editing tasks.

Output: A single generated image per prompt.

Scoring recipe

def evaluate(model, benchmark_name, test_set):
    scores = []
    for item in test_set:
        prompt = item['prompt']
        ref_img = item.get('reference_image')
        gen_img = model.generate(prompt, ref_img)
        score = get_benchmark_score(benchmark_name, gen_img, prompt, ref_img)
        scores.append(score)
    return sum(scores) / len(scores) if scores else 0.0

Common pitfalls

• Different sampling schedules or CFG scales drastically change scores on GenEval and DPGBench.
• Using unofficial or outdated versions of benchmark evaluators (e.g., GenEval's LLM-based script) leads to non-comparable results.
• Confusing SFT-only and RL-finetuned checkpoint results, as RL consistently yields higher scores across all benchmarks.

Evidence (verbatim from paper)

We assess general text-to-image generation using GenEval [48] to measure fundamental semantic alignment, and DPG-Bench [49] to assess long-prompt instruction following. In addition, we adopt UniGenBench [27] for a comprehensive and fine-grained evaluation of general generation capability, covering ten major categories (e.g., attribute binding, style control, and text rendering).

Citation

@misc{wang2026deepgen,
  title={DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.12205}
}

• arXiv: 2602.12205