Screen2words Eval

screen2words-eval

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning — Wang et al. (2021) (arXiv:2108.03353, 2021)

What this evaluates

Evaluates a model's ability to automatically generate concise, coherent language summaries of mobile UI screens by fusing visual, structural, and textual modalities. It probes multimodal representation learning and language generation capabilities in the context of human-computer interaction and UI understanding.

Datasets

• Screen2Words — total 22417; splits: train (-1), val (-1), test (-1)

Metrics

• BLEU-4 (primary) — range: [0, 100]
  • Computes modified 4-gram precision between the predicted summary and the reference set, combined with a brevity penalty to penalize overly short outputs.
• CIDEr — range: [0, 100]
  • Measures consensus-based image description evaluation using TF-IDF weighted n-gram cosine similarity between predictions and multiple human references.
• ROUGE-L — range: [0, 100]
  • Computes the F1 score based on the longest common subsequence (LCS) between the predicted and reference summaries, capturing sentence-level fluency.
• METEOR — range: [0, 100]
  • Matches unigrams based on exact, stem, synonym, and paraphrase matches, then computes a weighted harmonic mean of precision and recall with a penalty for fragmentation.

Input / output format

Input: Mobile UI screen image (processed via ResNet), structural layout data from the view hierarchy, screen text, and app description.

Output: A single text summary phrase generated sequentially via beam search (beam size 5).

Scoring recipe

def compute_metrics(predictions, gold_labels):
    # gold_labels: list of 5 reference phrases per screen
    scores = []
    for pred_tokens, refs in zip(predictions, gold_labels):
        # Remove UNK tokens from decoded phrase before evaluation
        pred_clean = [w for w in pred_tokens if w != '<UNK>']
        # Compute n-gram/LCS overlap against all 5 references
        bleu = compute_bleu(n=4, pred=pred_clean, refs=refs)
        cider = compute_cider(pred=pred_clean, refs=refs)
        rouge = compute_rouge_l(pred=pred_clean, refs=refs)
        meteor = compute_meteor(pred=pred_clean, refs=refs)
        scores.append((bleu + cider + rouge + meteor) / 4)
    return mean(scores) * 100  # Paper reports all metrics scaled by 100

Common pitfalls

• App-wise splitting is strictly enforced; failing to keep all screens from the same app within a single split causes severe data leakage and artificially inflates test performance.
• Each screen has 5 human-annotated reference summaries; evaluation must use all 5 as the reference set, not just a single randomly sampled label.
• UNK tokens in the model's decoded output are explicitly removed before metric calculation, which alters n-gram counts and must be handled during scoring.

Evidence (verbatim from paper)

In this section, we report our model performance based on metrics commonly used in machine translation and image captioning tasks: BLEU [34], CIDEr [43], ROUGE-L [29], and METOER [9] (see Table 2). A higher number means better model performance for these metrics—the closer distances between the predicted and the ground truth phrases.

Citation

@misc{wang2021screen2words,
  title={Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2108.03353}
}

• arXiv: 2108.03353