Word Learning Eval

word-learning-eval

Visual Grounding Helps Learn Word Meanings in Low-Data Regimes — Zhuang et al. (2023) (arXiv:2310.13257, 2023)

What this evaluates

Evaluates how well multi-modal models learn word meanings and semantic relationships from limited data, comparing visual grounding against language-only baselines across word-relatedness, feature prediction, and POS tagging tasks.

Datasets

• SimLex-999 — total ?; splits: test (-1)
• SimVerb-3500 — total ?; splits: test (-1)
• Word-relatedness dataset (Bruni et al., 2012) — total ?; splits: test (-1)

Metrics

• human-likeness measure (primary) — range: [0, 1]
  • Correlation between model-pair similarity scores and human similarity judgments. Pairs are ranked by the absolute difference between human and model similarity to compute a normalized human-likeness rank.
• prediction accuracy — range: [0, 1]
  • Standard classification accuracy for semantic feature prediction and POS tagging benchmarks.
• learning efficiency — range: percent
  • Model performance measured across varying data scales (e.g., 50M tokens, 2.1M image-caption pairs) to assess sample efficiency.

Input / output format

Input: Word pairs or single words paired with image features, full captions, or narrow textual contexts (e.g., 3-word segments). For sentence-level tasks, segmented sentences with center words and associated images.

Output: Model similarity scores for word pairs, predicted semantic features, POS tags, or probability distributions over words.

Scoring recipe

# Compute model similarity for word pairs
model_sim = cosine_similarity(model_emb[w1], model_emb[w2])
human_sim = human_judgments[w1, w2]
# Primary metric: correlation with human judgments
metric = spearmanr(model_sim, human_sim).correlation
# Secondary: accuracy for feature/POS prediction
acc = accuracy_score(gold_labels, predicted_labels)

Common pitfalls

• Visual grounding benefits are highly data-dependent; they disappear or reverse in high-data regimes where distributional text dominates.
• Models often fail to synergistically combine visual and textual signals, instead showing competitive interference.
• Visual models perform poorly on abstract words and verbs, as static images lack dynamic/action information.

Evidence (verbatim from paper)

To further analyze differences at the word level, we then develop a human-likeness measure for a pair of words by comparing the judgments of humans and models (see Appendix[A.2] for details). This measure is sorted to get a normalized “rank in human-likeness” measure for each pair of words, where a larger value means more human-like.

Citation

@misc{zhuang2023visualgrounding,
  title={Visual Grounding Helps Learn Word Meanings in Low-Data Regimes},
  author={Zhuang et al. (2023)},
  year={2023},
  note={arXiv:2310.13257}
}

• arXiv: 2310.13257