Gebench Eval

gebench-eval

GEBench: Benchmarking Image Generation Models as GUI Environments — Haodong Li et al. (2026) (arXiv:2602.09007, 2026)

What this evaluates

Evaluates image generation models' ability to function as dynamic GUI environments, probing temporal coherence, multi-step interaction logic, spatial grounding, and visual fidelity across sequential state transitions.

Datasets

• GEBench — total 700; splits: test (700); repo https://github.com/stepfun-ai/GEBench

Metrics

• GE-Score (primary) — range: [0, 100]
  • Composite score averaging five dimensions: Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Each dimension is scored by VLM judges, averaged across dimensions and tasks, then scaled to a 0–100 range.

Input / output format

Input: Initial GUI state image + textual instruction/prompt specifying the target interaction or transition.

Output: Generated GUI image(s) representing the resulting state after executing the instruction.

Scoring recipe

def compute_ge_score(predictions, gold):
    all_scores = []
    for traj in predictions:
        traj_scores = []
        for _ in range(3):
            dim_scores = [judge_score(traj, dim) for dim in ['Goal Achievement', 'Interaction Logic', 'Content Consistency', 'UI Plausibility', 'Visual Quality']]
            traj_scores.append(mean(dim_scores))
        all_scores.append(mean(traj_scores))
    return mean(all_scores) * 100

Common pitfalls

• Models excel at single-step transitions but suffer severe performance drops in multi-step planning due to error accumulation.
• VLM judges may inherit biases from their base models; the paper mitigates this by using three independent cross-evaluators and averaging results.
• Precise spatial grounding (e.g., mapping abstract [0,1000] coordinates to pixel locations) is a critical bottleneck often overlooked in standard image generation benchmarks.

Evidence (verbatim from paper)

To ensure the objectivity and robustness of GEBench, we deploy 3 state-of-the-art VLMs as independent cross-evaluators: 2 commercial models Gemini-3-Flash-Native, GPT-4o and 1 open-source model Qwen3-vl-235b-a22b-thinking. By utilizing these evaluators, we mitigate potential bias inherent in a single judge model. To ensure fair and reproducible comparisons, we use official default configurations for evaluated models and perform evaluation three times for each generated GUIs trajectory. Experimental results in Table [1] show that Nano Banana Pro delivers the most robust performance, particularly on Chinese subset with a top-ranking GE-Score of 69.62.

Citation

@misc{li2026gebench,
  title={GEBench: Benchmarking Image Generation Models as GUI Environments},
  author={Haodong Li et al. (2026)},
  year={2026},
  note={arXiv:2602.09007}
}

• arXiv: 2602.09007