Gui Ceval Eval

gui-ceval-eval

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents — Li et al. (2026) (arXiv:2603.15039, 2026)

What this evaluates

Evaluates multimodal large language models and agents on Chinese mobile GUI interaction tasks. It probes atomic capabilities like visual perception, grounding, and planning, as well as end-to-end execution reliability in both offline simulation and real-device online environments.

Datasets

• GUI-CEval — total ?; splits: test (-1)

Metrics

• Online Agent success rate (primary) — range: percent
  • Percentage of tasks completed successfully on real mobile devices. Measures end-to-end capability under real-world constraints.
• Step Success Rate (SSR) — range: percent
  • Percentage of individual steps executed correctly in offline execution traces. Computed as correct steps divided by total ground-truth steps.
• Grounding accuracy — range: percent
  • Percentage of correctly localized click targets. Measured only on samples where both annotation and prediction are clicks to avoid perception bias from mixed action types.
• Overall average score — range: percent
  • Aggregated percentage across five hierarchical capability dimensions: perception, planning, reflection, execution, and evaluation.

Input / output format

Input: Chinese language task instructions paired with mobile GUI screenshots (images). Screenshots may be provided at original resolution or scaled down to 1/2, 1/4, or 1/6 of original width/height.

Output: Model-generated action sequences (e.g., click coordinates, text input, scroll) and/or reasoning chains. Predictions are parsed to determine if the predicted action matches the ground-truth annotation for each step.

Scoring recipe

def compute_metrics(predictions, ground_truth, online_tasks):
    # Grounding Accuracy (clicks only)
    click_pairs = [(p, g) for p, g in zip(predictions, ground_truth) if g.action == 'click']
    grounding_acc = sum(1 for p, g in click_pairs if p.action == 'click' and match_coords(p, g)) / len(click_pairs) * 100
    
    # Step Success Rate (SSR)
    ssr = sum(1 for p, g in zip(predictions, ground_truth) if actions_match(p, g)) / len(ground_truth) * 100
    
    # Online Agent Success Rate
    for task in online_tasks if task.completed_on_device) / len(online_tasks) * 100
    
    return {'grounding_accuracy': grounding_acc, 'ssr': ssr, 'online_success_rate': online_success}

Common pitfalls

• Grounding accuracy is strictly limited to click actions to avoid perception bias from mixed action types; evaluating it on all actions inflates or deflates the metric unfairly.
• Online success rate exhibits a cliff-like decline beyond 6 steps, making short-horizon or offline metrics poor proxies for real-world deployment capability.
• Initial page state heavily influences results; evaluating only from the home screen overestimates robustness for uncontrolled user paths and diverse app entry points.

Evidence (verbatim from paper)

When examined by capability dimension, perception-related scores such as Perception and Grounding remain high (peaking at 82.28% and 90.10%, respectively), while reasoning and correction-related metrics (Reflection and Evaluation) lag far behind (maximums of 21.01% and 49.24%). For Online Agent—the most challenging task involving real interactions—the best success rate is 33.33% (UI-TARS-72B-SFT), with over half of the models below 20%.

Citation

@misc{li2026guiceval,
  title={GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.15039}
}

• arXiv: 2603.15039