Ghostui Eval

ghostui-eval

GhostUI: Unveiling Hidden Interactions in Mobile UI — Kweon et al. (2026) (arXiv:2601.19258, 2026)

What this evaluates

This benchmark evaluates vision-language models' ability to detect and predict hidden interactions in mobile user interfaces. It probes whether models can infer concealed gestures (e.g., long presses, swipes) from before-interaction screenshots and task descriptions, localize the interaction target, and anticipate the resulting UI state changes.

Datasets

• GhostUI — total ?; splits: train (-1), test (-1); repo https://github.com/gh0stui/ghostui

Metrics

• Accuracy (primary) — range: percent
  • Percentage of instances where the model correctly predicts the gesture type, including directional specifications for swipes, scrolls, and pinches.
• IoU — range: [0, 1]
  • Intersection over Union between the predicted bounding box and the ground truth bounding box, measuring spatial localization precision.
• cosine similarity — range: [-1, 1]
  • Cosine similarity between the text embedding of the model's predicted UI description and the ground truth description, computed using Google's gemini-embedding-001 model.

Input / output format

Input: Before-interaction screenshot, task description, and optionally simplified view hierarchy, gesture usage patterns, and app metadata.

Output: Predicted gesture type and bounding box coordinates (Task 1); text description of the post-gesture UI state (Task 2).

Scoring recipe

def score_task1(pred_gesture, pred_box, gold_gesture, gold_box):
    acc = 1.0 if pred_gesture == gold_gesture else 0.0
    iou = intersection_area(pred_box, gold_box) / union_area(pred_box, gold_box)
    return acc, iou

def score_task2(pred_text, gold_text):
    pred_emb = gemini_embed_001(pred_text)
    gold_emb = gemini_embed_001(gold_text)
    return cosine_similarity(pred_emb, gold_emb)

Common pitfalls

• Models often memorize app-specific patterns rather than learning generalizable gesture rules due to the app-level train/test split.
• Zero-shot models exhibit a strong bias toward predicting 'tap' gestures, frequently misclassifying double taps and long presses as simple taps.
• Spatial localization (IoU) is highly sensitive to the absence of view hierarchy data, causing drastic performance drops even if gesture classification remains stable.
• Double tap outcomes are highly context-dependent and diverse, making them consistently difficult to predict accurately even after fine-tuning.

Evidence (verbatim from paper)

First, classification accuracy measures whether the model correctly predicts the gesture type, including directional specifications for swipes (left, right), scrolls (up, down), and pinch gestures (zoom in, zoom out). Second, Intersection over Union (IoU) quantifies the overlap between predicted and ground truth bounding boxes to assess spatial localization precision. For [Section 4.3], we measured the model’s understanding of interaction outcomes by calculating cosine similarity between predicted UI descriptions and ground truth descriptions generated from after-interaction screenshots.

Citation

@misc{kweon2026ghostui,
  title={GhostUI: Unveiling Hidden Interactions in Mobile UI},
  author={Kweon et al. (2026)},
  year={2026},
  note={arXiv:2601.19258}
}

• arXiv: 2601.19258