Gui Agent Halluc Eval

gui-agent-halluc-eval

HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents — Chao Jin et al. (2026) (arXiv:2604.17284, 2026)

What this evaluates

This protocol evaluates GUI agents on visual grounding, action execution, and hallucination rates across mobile, desktop, and web interfaces. It measures how well models localize UI elements, execute multi-step tasks under varying instruction granularities, and avoid perception or reasoning errors.

Datasets

• ScreenSpot-V2 — total ?; splits: test (-1)
• ScreenSpot-Pro — total ?; splits: test (-1)
• AndroidControl — total ?; splits: test (-1)
• GUI-Odyssey — total ?; splits: test (-1)

Metrics

• Action Type Accuracy (Type) (primary) — range: percent
  • Exact match accuracy between the predicted and ground-truth action types.
• Grounding Accuracy (GR) (primary) — range: percent
  • Evaluates the precise localization of click points within specific action categories (e.g., CLICK).
• Step-wise Success Rate (SR) (primary) — range: percent
  • Quantifies alignment between prediction and ground truth by calculating the intersection of Type and GR.
• Hallucination Rate (HR) (primary) — range: percent
  • Overall rate of hallucination behaviors classified by VLM judges, normalized by the judge's credibility score to mitigate bias.

Input / output format

Input: GUI screenshot/image paired with a text instruction (global high-level or low-level with pre-annotated step-by-step actions).

Output: Predicted action type, grounding coordinates (click point), and reasoning/trajectory steps.

Scoring recipe

def compute_metrics(predictions, golds):
    type_acc = sum(1 for p, g in zip(predictions, golds) if p.type == g.type) / len(golds)
    gr_acc = sum(1 for p, g in zip(predictions, golds) if dist(p.click, g.click) <= THRESHOLD) / len(golds)
    sr = type_acc * gr_acc
    hr = 1.0 - (sum(1 for p in predictions if not p.hallucination) / len(predictions))
    return {'Type': type_acc, 'GR': gr_acc, 'SR': sr, 'HR': hr}

Common pitfalls

• Low-level vs high-level difficulty settings differ in input granularity; confusing them leads to incorrect capability assessment.
• HR is normalized by VLM judge credibility scores; ignoring this calibration inflates or deflates hallucination rates depending on judge bias.
• GR requires a spatial threshold for click point localization that is not explicitly defined in the provided text, requiring external benchmark documentation.

Evidence (verbatim from paper)

Specifically, we employ three primary metrics: Action Type Accuracy (Type), which measures the exact match accuracy between the predicted and ground-truth action types; Grounding Accuracy (GR), which evaluates the precise localization of click points within specific action categories (e.g., CLICK); and Step-wise Success Rate (SR), which quantifies the alignment between prediction and ground truth by calculating the intersection of Type and GR.

Citation

@misc{jin2026halluclear,
  title={HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents},
  author={Chao Jin et al. (2026)},
  year={2026},
  note={arXiv:2604.17284}
}

• arXiv: 2604.17284