# Uibert UI Understanding Eval

> This evaluation probes a model's ability to understand and reason about user interface components across multiple modalities (images, text, structural metadata). It tests cross-modal alignment, component retrieval, synchronization detection, and classification tasks relevant to UI design and accessibility. Use when the user wants to benchmark on Rico, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/uibert-ui-understanding-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/uibert-ui-understanding-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/uibert-ui-understanding-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/uibert-ui-understanding-eval

---


# uibert-ui-understanding-eval

> UIBert: Learning Generic Multimodal Representations for UI Understanding — Bai et al. (2021) (arXiv:2107.13731, 2021)

## What this evaluates

This evaluation probes a model's ability to understand and reason about user interface components across multiple modalities (images, text, structural metadata). It tests cross-modal alignment, component retrieval, synchronization detection, and classification tasks relevant to UI design and accessibility.

## Datasets

- **Rico** — total 1000000; splits: train (900000), dev (32000), test (32000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted labels or retrieved candidates out of the total number of instances.
- `macro-F1` — range: other
  - Unweighted mean of the F1 score calculated independently for each class, then averaged across all classes.

## Input / output format

**Input**: UI components represented as images (IMG), OCR text, and structural view hierarchy (VH) metadata. For retrieval tasks, an anchor UI/component and a search UI with candidate components. For classification/sync tasks, full UI embeddings or concatenated component embeddings.

**Output**: For retrieval: a single selected candidate component index. For classification/sync: a predicted class label (e.g., app type, icon type, sync/unsync).

## Scoring recipe

```python
def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    classes = sorted(set(gold + predictions))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return acc, sum(f1s) / len(f1s)
```

## Common pitfalls

- Web UI evaluation lacks view hierarchy (VH) components, forcing models to rely only on image and OCR text, which drastically changes the input modality compared to mobile UIs.
- Zero-shot evaluation uses the pretrained model directly without task-specific finetuning, so results test generalization rather than optimized task performance.
- App type classification results differ from prior work (ActionBert) because this paper uses the full 72k Rico dataset instead of a 43.5k subset.

## Evidence (verbatim from paper)

> Overall, prediction accuracy of all methods on the four task variations are reported in Tab. [1]. We observe that UIBert outperforms both baselines on all cases by 0.85%–9.26%, especially by a large margin on the zero-shot tasks.

## Citation

```bibtex
@misc{bai2021uibert,
  title={UIBert: Learning Generic Multimodal Representations for UI Understanding},
  author={Bai et al. (2021)},
  year={2021},
  note={arXiv:2107.13731}
}
```

- arXiv: 2107.13731