qhjqhj00 screenpr-eval
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding — Fan et al. (2024) (arXiv:2406.19263, 2024)
What this evaluates
Evaluates a model's ability to read and describe the content and layout of a GUI screenshot at a specific pointed location. It probes layout-aware screen reading, spatial reasoning, and the capacity to generate focused descriptions for mobile agent navigation.
Datasets
- ScreenPR — total ?; splits: test (-1)
Metrics
Content Acc(primary) — range: percent- Accuracy of the generated content description compared to human-verified ground truth descriptions.
Layout Acc— range: percent- Accuracy of the generated layout description compared to human-verified ground truth descriptions.
F1 score— range: [0, 1]- Harmonic mean of precision and recall for verifying whether a mobile navigation agent's action is correct or incorrect.
Input / output format
Input: A GUI screenshot $S_i$ and a text prompt containing a specific point coordinate $P_i$.
Output: Text descriptions detailing the content and layout of the GUI region surrounding the pointed location.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = 0
for pred, gold_desc in zip(predictions, gold):
if pred == gold_desc: # Exact match or human-verified label match
correct += 1
return (correct / len(gold)) * 100
def compute_f1(tp, fp, fn):
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Language similarity metrics like BERTScore and ROUGE-L often fail to distinguish model performance because generated captions can be comprehensive but redundant, whereas focused screen-reading outputs are preferred by humans.
- Human evaluation preferences may diverge from automatic accuracy scores; models producing longer, more comprehensive descriptions may score lower on accuracy but be preferred by human judges.
Evidence (verbatim from paper)
We evaluate our ToL agent against three baselines on the ScreenPR benchmark as shown in Table [2]. The results are consistent across both human evaluation and automatic cycle consistency evaluation, showing that our ToL agent achieves the best performance in terms of content and layout description accuracy.
Citation
@misc{fan2024readanywherepointed,
title={Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding},
author={Fan et al. (2024)},
year={2024},
note={arXiv:2406.19263}
}
- arXiv: 2406.19263