Screendrag Eval

screendrag-eval

Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging — Liao et al. (2025) (arXiv:2601.06031, 2025)

What this evaluates

Evaluates a model's ability to perform fine-grained text dragging interactions on GUI screenshots. It measures whether the model correctly triggers a drag action, accurately selects the target text span, and aligns its predicted coordinates with ground truth.

Datasets

• SCREENDRAG — total 5333; splits: test (5333)

Metrics

• DTR (primary) — range: percent
  • Drag Trigger Rate: the percentage of test cases where the model outputs a drag action instead of a click or other action.
• B-Dist — range: pixels
  • Bounding Box Distance: the distance between the predicted and ground-truth bounding box coordinates for the dragged text span. Lower values indicate better alignment.
• SR — range: percent
  • Span Recall: the fraction of ground-truth text spans correctly predicted by the model. A prediction is considered correct if the distance to the ground truth is within a threshold φ = 3 pixels.

Input / output format

Input: A GUI screenshot image paired with a natural language instruction specifying a text dragging action (e.g., 'drag the word X'). Instructions vary in text density (sparse vs. dense) and interface context levels.

Output: A drag action defined by start and end coordinates (or bounding boxes) for the target text span.

Scoring recipe

def compute_metrics(predictions, golds):
    dtr = sum(1 for p in predictions if p.action == 'drag') / len(predictions)
    valid_preds = [p for p in predictions if p.action == 'drag']
    valid_golds = [g for g in golds if g.action == 'drag']
    b_dists, sr_scores = [], []
    for p, g in zip(valid_preds, valid_golds):
        dist = bbox_distance(p.coords, g.coords)
        b_dists.append(dist)
        sr_scores.append(1 if dist <= 3 else 0)
    return dtr, mean(b_dists), mean(sr_scores)

Common pitfalls

• B-Dist and SR are only computed on cases where the model successfully outputs a drag action, which masks failures to trigger the drag entirely.
• Models exhibit a strong bias toward clicking, often ignoring explicit drag instructions and defaulting to click actions even when prompted.
• The SR threshold of 3 pixels is empirically derived from 100 manual inspections and may not generalize across different screen resolutions or font sizes.

Evidence (verbatim from paper)

We evaluate models using the three metrics introduced in Section 3, each designed to capture different aspects of text dragging performance. For the B-Dist and SR metrics, we only consider cases where the model can accurately output the drag action. For the SR metric, we set the threshold φ to 3 pixels. This value is empirically determined by manually inspecting 100 examples, and is found to strike a good balance between reducing false positives and false negatives.

Citation

@misc{liao2025beyondclicking,
  title={Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging},
  author={Liao et al. (2025)},
  year={2025},
  note={arXiv:2601.06031}
}

• arXiv: 2601.06031