Uibert UI Understanding Eval

uibert-ui-understanding-eval

UIBert: Learning Generic Multimodal Representations for UI Understanding — Bai et al. (2021) (arXiv:2107.13731, 2021)

What this evaluates

This evaluation probes a model's ability to understand and reason about user interface components across multiple modalities (images, text, structural metadata). It tests cross-modal alignment, component retrieval, synchronization detection, and classification tasks relevant to UI design and accessibility.

Datasets

• Rico — total 1000000; splits: train (900000), dev (32000), test (32000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels or retrieved candidates out of the total number of instances.
• macro-F1 — range: other
  • Unweighted mean of the F1 score calculated independently for each class, then averaged across all classes.

Input / output format

Input: UI components represented as images (IMG), OCR text, and structural view hierarchy (VH) metadata. For retrieval tasks, an anchor UI/component and a search UI with candidate components. For classification/sync tasks, full UI embeddings or concatenated component embeddings.

Output: For retrieval: a single selected candidate component index. For classification/sync: a predicted class label (e.g., app type, icon type, sync/unsync).

Scoring recipe

def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    classes = sorted(set(gold + predictions))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return acc, sum(f1s) / len(f1s)

Common pitfalls

• Web UI evaluation lacks view hierarchy (VH) components, forcing models to rely only on image and OCR text, which drastically changes the input modality compared to mobile UIs.
• Zero-shot evaluation uses the pretrained model directly without task-specific finetuning, so results test generalization rather than optimized task performance.
• App type classification results differ from prior work (ActionBert) because this paper uses the full 72k Rico dataset instead of a 43.5k subset.

Evidence (verbatim from paper)

Overall, prediction accuracy of all methods on the four task variations are reported in Tab. [1]. We observe that UIBert outperforms both baselines on all cases by 0.85%–9.26%, especially by a large margin on the zero-shot tasks.

Citation

@misc{bai2021uibert,
  title={UIBert: Learning Generic Multimodal Representations for UI Understanding},
  author={Bai et al. (2021)},
  year={2021},
  note={arXiv:2107.13731}
}

• arXiv: 2107.13731