Guicourse Gui Nav Eval

guicourse-gui-nav-eval

GUICourse: From General Vision Language Models to Versatile GUI Agents — Chen et al. (2024) (arXiv:2406.11317, 2024)

What this evaluates

Evaluates vision-language models' ability to navigate graphical user interfaces by predicting correct action types and precise screen coordinates. It probes OCR, pixel-level grounding, and multi-step task planning across web and mobile environments.

Datasets

• GUIAct — total 4400; splits: test (4400)
• Mind2Web — total 2000; splits: test (2000)
• AITW — total ?; splits: test (-1)

Metrics

• StepSR (primary) — range: percent
  • Average success rate across all actions in a trajectory. Calculated as the mean of per-action success rates.
• Type EM — range: percent
  • Accuracy of predicting the correct action type (e.g., click, swipe, scroll).
• Cli.Acc — range: percent
  • Average success rate specifically for 'click' and 'tap' actions, reflecting pixel-level grounding ability.
• Ele.Acc — range: percent
  • Accuracy of predicting correct position parameters for actions, used primarily on Mind2Web.

Input / output format

Input: Screenshot/image of the GUI interface and a natural language instruction (optionally with multi-turn dialogue history).

Output: A structured action command specifying the action type (e.g., 'click', 'swipe') and its target coordinates (e.g., x y or bounding box).

Scoring recipe

def compute_metrics(pred_traj, gold_traj):
    n = len(gold_traj)
    type_em = sum(1 for p, g in zip(pred_traj, gold_traj) if p['type'] == g['type']) / n
    click_steps = [g for g in gold_traj if g['type'] in ['click', 'tap']]
    cli_acc = sum(1 for p, g in zip(pred_traj, gold_traj) if g['type'] in ['click', 'tap'] and iou(p['bbox'], g['bbox']) > 0.5) / max(1, len(click_steps))
    step_sr = sum(1 for p, g in zip(pred_traj, gold_traj) if p['type'] == g['type'] and iou(p['bbox'], g['bbox']) > 0.5) / n
    return type_em, cli_acc, step_sr

Common pitfalls

• Static evaluation assumes a single golden trajectory, causing 'false errors' when multiple valid action sequences exist for the same instruction.
• Performance heavily depends on input resolution; models trained at 448x448 significantly underperform compared to 1344x1344 versions on fine-grained grounding tasks.
• Evaluations on AITW and Mind2Web use processed subsets from SeeClick, excluding the 'General' column for AITW to avoid data leakage with the training GUIAct dataset.

Evidence (verbatim from paper)

We use the type exact match score (Type EM), click accuracy (Cli. Acc), and step success rate (StepSR) as metrics. Type EM is the accuracy of the actions’ names. Cli.Acc is the average score of the success rate of the "click" and "tap" actions, which reflects the grounding ability. StepSR is the average score of all the actions’ success rates.

Citation

@misc{chen2024guicourse,
  title={GUICourse: From General Vision Language Models to Versatile GUI Agents},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.11317}
}

• arXiv: 2406.11317