Word Embedding Eval

word-embedding-eval

Visual Grounding of Inter-lingual Word-Embeddings — Mohammed et al. (2022) (arXiv:2209.03714, 2022)

What this evaluates

Evaluates the quality of multilingual word embeddings by measuring semantic similarity/relatedness and word categorization accuracy. It probes whether visual grounding improves cross-lingual semantic alignment and clustering of basic-level concepts.

Datasets

• WordSim353 — total ?; splits: test (-1)
• MEN — total ?; splits: test (-1)
• RW — total ?; splits: test (-1)
• MTurk — total ?; splits: test (-1)
• simVerb — total ?; splits: test (-1)
• SimLex999 — total ?; splits: test (-1)
• Battig — total ?; splits: test (-1)
• AP — total ?; splits: test (-1)
• BLESS — total ?; splits: test (-1)
• ESSLLI-a — total ?; splits: test (-1)
• ESSLLI-b — total ?; splits: test (-1)
• ESSLLI-c — total ?; splits: test (-1)
• Almarsoomi — total ?; splits: test (-1)
• MC30 — total ?; splits: test (-1)
• Saif40 — total ?; splits: test (-1)
• WordSim — total ?; splits: test (-1)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Rank-based correlation between predicted similarity/relatedness scores and human-annotated gold scores.
• Purity score — range: [0, 1]
  • Sum of the size of the largest cluster for each gold category, divided by the total number of instances.

Input / output format

Input: Word pairs with gold similarity/relatedness scores; sets of nouns with gold category labels. Visual grounding uses image-caption pairs from COCO.

Output: Predicted similarity/relatedness scores for word pairs; predicted cluster assignments for categorization tasks.

Scoring recipe

# Similarity/Relatedness
spearman_rho = spearmanr(gold_scores, pred_scores).correlation

# Categorization (Purity)
purity = 0
for cluster in unique_clusters:
    cluster_labels = gold_labels[cluster]
    purity += max(Counter(cluster_labels).values())
purity /= len(gold_labels)

Common pitfalls

• Confusing word similarity (is-a hierarchy) with word relatedness (association), which some benchmarks conflate.
• Using Pearson correlation instead of Spearman rank correlation for similarity benchmarks, which changes the evaluation metric.
• Not accounting for the fact that BLESS scores are normalized per concept before averaging across relations.

Evidence (verbatim from paper)

Following (Bruni et al., 2014; Shahmohammadi et al., 2022), we evaluated our visually grounded word embeddings using similarity/relatedness benchmarks. The task is to estimate the similarity/relatedness score of a pair of words using the Spearman correlation as evaluation metric.

Citation

@misc{mohammed2022visualgrounding,
  title={Visual Grounding of Inter-lingual Word-Embeddings},
  author={Mohammed et al. (2022)},
  year={2022},
  note={arXiv:2209.03714}
}

• arXiv: 2209.03714