Guiodyssey Eval

guiodyssey-eval

GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices — Quanfeng Lu et al. (2024) (arXiv:2406.08451, 2024)

What this evaluates

Evaluates multimodal agents' ability to perform cross-app GUI navigation on mobile devices by predicting correct UI actions based on screen states and task instructions. It probes spatial reasoning, action planning, and the model's capacity to leverage historical context across multiple applications.

Datasets

• GUIOdyssey — total 8334; splits: Train-Random (-1), Test-Random (-1), Train-App (-1), Test-App (-1), Train-Task (-1), Test-Task (-1), Train-Device (-1), Test-Device (-1); repo https://github.com/OpenGVLab/GUI-Odyssey

Metrics

• Action Matching Score (AMS) (primary) — range: percent
  • Percentage of correctly predicted actions across a dataset or episode. An action is correct if: (1) action type matches ground truth; (2) for CLICK/LONG PRESS, predicted coordinates fall within 14% of screen distance from reference; (3) for SCROLL, direction matches; (4) for TYPE, Average Normalized Levenshtein Similarity (ANLS) < 0.5.
• Success Rate (SR) — range: percent
  • Episode-level success rate. A task/episode is counted as successful only if every single action within it is predicted correctly. AMS must be 100% for the episode to count as a success.

Input / output format

Input: Current screen screenshot, task instruction (high-level or low-level), and optional historical context (previous actions, historical screenshots, and/or semantic annotations summarizing past steps).

Output: A sequence of GUI actions per step, specifying action type (CLICK, LONG PRESS, SCROLL, TYPE), target coordinates or scroll direction, and text string for TYPE actions.

Scoring recipe

def compute_ams(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.type != gold.type: continue
        if pred.type in ['CLICK', 'LONG_PRESS']:
            if dist(pred.coords, gold.coords) <= 0.14 * screen_size: correct += 1
        elif pred.type == 'SCROLL':
            if pred.direction == gold.direction: correct += 1
        elif pred.type == 'TYPE':
            if anls(pred.text, gold.text) < 0.5: correct += 1
    return (correct / len(golds)) * 100

def compute_sr(predictions, golds):
    success_count = 0
    for ep_preds, ep_golds in zip(predictions, golds):
        if compute_ams(ep_preds, ep_golds) == 100: success_count += 1
    return (success_count / len(golds)) * 100

Common pitfalls

• Success Rate (SR) is extremely strict: an entire episode fails if even one action is incorrect, making it disproportionately low for long-step tasks.
• Coordinate matching uses a 14% screen-distance threshold rather than exact pixel matching, and relies on SAM2 segmentation for target element validation.
• TYPE actions are evaluated using ANLS < 0.5, not exact string equality, which can mask minor typographical errors.

Evidence (verbatim from paper)

Evaluation Metrics. To ensure reproducibility and efficiency, we adopt an offline evaluation method to benchmark performance. We use the Action Matching Score (AMS) as our metric, inspired by the approaches presented in AITW and AutoUI. An action is considered correct if its action type matches the ground-truth type. Additionally, for CLICK and LONG PRESS actions, we consider them correct if they fall within 14% of the screen distance from the reference gesture. Furthermore, we utilize SAM2 to determine the coordinates of the target element, and if the predicted coordinates lie within the region segmented by SAM2, the action is also deemed correct. As for SCROLL actions, we compare whether the direction (i.e., up, down, left, or right) matches the gold gesture’s direction. For TYPE actions, we evaluate the Average Normalized Levenshtein Similarity (ANLS) between the predicted and gold gestures. If the ANLS is below a certain threshold (set to 0.5 in our experiments), we consider it correct. We then calculate Success Rate (SR) for the whole episode. A task is considered successful only if all actions are correct. Success Rate (SR) is a rigorous metric. It would be harder to achieve

Citation

@misc{lu2024guiodyssey,
  title={GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices},
  author={Quanfeng Lu et al. (2024)},
  year={2024},
  note={arXiv:2406.08451}
}

• arXiv: 2406.08451