Winoground Eval

winoground-eval

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality — Thrush et al. (2022) (arXiv:2204.03162, 2022)

What this evaluates

Probes vision-language models' ability to understand visio-linguistic compositionality and word order sensitivity. The task requires matching images to captions where identical words are rearranged to change the described scene, testing structural grounding rather than lexical overlap.

Datasets

• Winoground — total 1600; splits: test (1600)

Metrics

• image-caption score (primary) — range: [0, 1]
  • Computed as the ratio of positive matches (either crowdworker votes or model predictions) to the total number of image-caption pairs. For models, typically derived from accuracy on the binary matching task or similarity scores.

Input / output format

Input: One image and one caption presented together as a pair.

Output: Binary choice: whether the caption matches the image (match/mismatch), or a similarity score used to derive the match decision.

Scoring recipe

def compute_image_caption_score(predictions, gold):
    # predictions: list of binary match/mismatch or similarity scores
    # gold: list of binary labels (1=match, 0=mismatch)
    correct = sum(1 for p, g in zip(predictions, gold) if (p == g) or (p > 0.5 and g == 1))
    return correct / len(gold)

Common pitfalls

• Models often exploit visual salience or caption length instead of compositional structure, leading to chance-level performance on carefully controlled pairs.
• All pairs in a test item share the exact same set of words, so models cannot rely on lexical overlap to distinguish correct matches from incorrect ones.
• The human baseline is calculated as a ratio of crowdworker votes rather than a simple majority vote, which can yield different scores than standard accuracy.

Evidence (verbatim from paper)

Annotators are asked the binary choice question “Does the caption match the image?”. All 1600 combinations of images and captions are labeled by at least ten annotators. We compute the human image-caption score as the ratio of annotators who said the image/caption pair match over the total number of annotators for the pair.

Citation

@misc{thrush2022winoground,
  title={Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality},
  author={Thrush et al. (2022)},
  year={2022},
  note={arXiv:2204.03162}
}

• arXiv: 2204.03162