Groundnext Eval

groundnext-eval

Grounding Computer Use Agents on Human Demonstrations — Feizi et al. (2025) (arXiv:2511.07332, 2025)

What this evaluates

Evaluates vision-language models on UI element localization and grounding across desktop, mobile, and web interfaces. It measures how accurately a model can identify and locate specific UI components based on text instructions, and assesses their effectiveness in multi-step agentic tasks.

Datasets

• SSPro — total ?; splits: test (-1)
• OSW-G — total ?; splits: test (-1)
• MMB-GUI — total ?; splits: test (-1)
• SSv2 — total ?; splits: test (-1)
• UI-V — total ?; splits: test (-1)
• OSWorld-Verified — total 361; splits: test (361)

Metrics

• average performance (primary) — range: percent
  • Arithmetic mean of accuracy scores across the listed benchmarks (SSPro, OSW-G, MMB-GUI, SSv2, UI-V). Reported as a percentage.

Input / output format

Input: Screen capture (screenshot) of a desktop, mobile, or web interface paired with a natural language instruction describing the target UI element or task.

Output: Bounding box coordinates or UI element identifier corresponding to the target described in the instruction.

Scoring recipe

def compute_avg_accuracy(predictions, golds):
    correct = 0
    total = 0
    for pred, gold in zip(predictions, golds):
        if match(pred, gold): # IoU > threshold or exact element match
            correct += 1
        total += 1
    return (correct / total) * 100

Common pitfalls

• The 'average performance' metric calculation varies across tables; Figure 3 explicitly excludes UI-Vision, while Table 2 includes it in the average.
• Agentic evaluation (OSWorld-Verified) relies on a specific external planner (o3) and a fixed Ubuntu 1920x1080 Docker environment, limiting direct comparability with models evaluated in different OS/resolution settings.
• RL post-training yields only marginal gains (~1-2 points) over high-quality SFT, so reporting RL-only results without SFT baselines can be misleading.

Evidence (verbatim from paper)

Our models achieve the highest average performance for both $3$B and $7$B model sizes.

Citation

@misc{feizi2025grounding,
  title={Grounding Computer Use Agents on Human Demonstrations},
  author={Feizi et al. (2025)},
  year={2025},
  note={arXiv:2511.07332}
}

• arXiv: 2511.07332