Widget Captioning Eval

widget-captioning-eval

Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements — Yang Li et al. (arXiv:2010.04295, 2020)

What this evaluates

This benchmark evaluates a model's ability to generate natural language descriptions for individual mobile UI elements using multimodal inputs. It probes the capability to fuse visual appearance and structural hierarchy data to produce accurate, context-aware captions for accessibility and UI understanding tasks.

Datasets

• Widget Captioning Dataset — total 61285; splits: train (52178), val (4548), test (4559); repo https://github.com/google-research/google-research/tree/master/widget-caption

Metrics

• CIDEr (primary) — range: [0, 1]
  • TF-IDF weighted n-gram similarity between predicted and reference captions. Higher scores indicate better alignment with ground truth.
• BLEU-1 — range: [0, 1]
  • Unigram precision with brevity penalty to penalize overly short predictions.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence recall-based overlap between prediction and reference.
• METEOR — range: [0, 1]
  • Alignment-based metric combining precision, recall, and synonymy matching with a penalty for fragmentation.
• SPICE — range: [0, 1]
  • Semantic parser-based metric that matches objects and relations in the predicted caption to the reference.

Input / output format

Input: Visual image of a target UI element (processed via ResNet) and structural representation including widget properties, local view hierarchy, and full screen context.

Output: A natural language caption sequence generated from a 10,000-word vocabulary. During evaluation, any tokens in the decoded output are removed before scoring.

Scoring recipe

def compute_metrics(predictions, references):
    # predictions: list of str, references: list of list of str
    # Strip <UNK> tokens as per paper protocol
    preds_clean = [p.replace('<UNK>', '').strip() for p in predictions]
    bleu1 = nltk.bleu_score.corpus_bleu(references, [p.split() for p in preds_clean], weights=(1,0,0,0))
    cider = cider_score.compute_score(references, preds_clean)
    rouge = rouge_score.rouge_l(references, preds_clean)
    meteor = meteor_score.compute_score(references, preds_clean)
    spice = spice_score.compute_score(references, preds_clean)
    return {'BLEU-1': bleu1, 'CIDEr': cider, 'ROUGE-L': rouge, 'METEOR': meteor, 'SPICE': spice}

Common pitfalls

• Splits are app-wise, not screen-wise: all screens from a given app are kept in a single split. Models must generalize to entirely unseen apps, not just unseen UI layouts.
• Multiple ground-truth captions per element: each widget has several reference captions. Metrics are computed against the full reference set, not a single gold caption.
• UNK token removal: any in the model's output is stripped before evaluation, which can artificially inflate n-gram overlap if not handled identically to the paper.

Evidence (verbatim from paper)

We report our accuracy based on BLEU (unigram and bigram), CIDEr, ROUGE-L, METOER and SPICE metrics (see Table 3). For all these metrics, a higher number means better captioning accuracy—the closer distances between the predicted and the groundtruth captions.

Citation

@misc{li2020widgetcaptioning,
  title={Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements},
  author={Yang Li et al.},
  year={2020},
  note={arXiv:2010.04295}
}

• arXiv: 2010.04295