Step Gui Eval

step-gui-eval

Step-GUI Technical Report — Yan et al. (2025) (arXiv:2512.15431, 2025)

What this evaluates

Evaluates a vision-language model's ability to perceive, locate, and interact with graphical user interfaces across desktop and mobile environments. It also measures general multimodal reasoning and OCR capabilities to ensure the model retains broad foundational skills after GUI-specific training.

Datasets

• ScreenSpot-Pro — total ?; splits: test (-1)
• ScreenSpot-v2 — total ?; splits: test (-1)
• OSWorld-G — total ?; splits: test (-1)
• MMBench-GUI-L2 — total ?; splits: test (-1)
• VisualWebBench — total ?; splits: test (-1)
• OSWorld-Verified — total 369; splits: test (369)
• AndroidWorld — total 116; splits: test (116)
• AndroidDaily — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted UI element locations or action types out of total instances. Calculated as correct predictions divided by total test samples.
• pass@1 — range: percent
  • Fraction of tasks where the model's first attempt exactly matches the gold standard action sequence or outcome.
• pass@3 (primary) — range: percent
  • Fraction of tasks where at least one of the model's first three attempts matches the gold standard. Used to mitigate environment instability.
• end-to-end success rate — range: percent
  • Percentage of complex, multi-step tasks completed successfully in live emulator environments without manual intervention.

Input / output format

Input: UI screenshots or screen recordings paired with natural language instructions or task prompts.

Output: Bounding box coordinates or element identifiers for grounding tasks; sequential action commands (click, type, slide, etc.) for agent tasks; text responses for multimodal benchmarks.

Scoring recipe

def compute_pass_k(predictions, gold, k=3):
    scores = []
    for attempts, target in zip(predictions, gold):
        scores.append(1.0 if any(a == target for a in attempts[:k]) else 0.0)
    return sum(scores) / len(scores)

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Environment instability (VM crashes, CAPTCHAs, ADB failures) necessitates using pass@3 instead of pass@1 for end-to-end benchmarks to avoid penalizing infrastructure faults.
• Static action prediction accuracy on AndroidDaily is reported separately from end-to-end task completion success rates, requiring careful distinction between per-action and per-task metrics.
• Chinese-language mobile apps in AndroidDaily require region-specific UI knowledge not covered by general English-centric benchmarks.

Evidence (verbatim from paper)

Due to the inherent instability of the OSWorld-Verified testing environment (including frequent VM crashes, substantial loading delays, and CAPTCHA interruptions), we employ the Pass@3 metric to mitigate failures caused by infrastructure issues rather than model limitations.

Citation

@misc{yan2025stepgui,
  title={Step-GUI Technical Report},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2512.15431}
}

• arXiv: 2512.15431