Retrofitting Word Vectors Eval

retrofitting-word-vectors-eval

Retrofitting Word Vectors to Semantic Lexicons — Faruqui et al. (2014) (arXiv:1411.4166, 2014)

What this evaluates

Evaluates the semantic quality of pre-trained word vectors by measuring performance improvements after applying a graph-based retrofitting method using semantic lexicons. It probes the model's ability to capture lexical relations (e.g., synonymy, hyponymy) and generalizes across different vector training methods, lexicon types, and languages.

Datasets

• MEN-3k — total ?; splits: test (-1)
• RG-65 — total ?; splits: test (-1)
• WS-353 — total ?; splits: test (-1)
• TOEFL — total ?; splits: test (-1)
• SYN-REL — total ?; splits: test (-1)
• SA — total ?; splits: test (-1)
• MC-30 — total ?; splits: test (-1)

Metrics

• Spearman's correlation (primary) — range: [0, 1]
  • Ranks the predicted similarity scores and gold standard scores, then computes the Pearson correlation coefficient between the two rank arrays. Measures monotonic relationship between predicted and human similarity judgments.
• Accuracy — range: [0, 1]
  • Proportion of correctly predicted labels or analogies out of the total number of instances. Used for classification and relation tasks.

Input / output format

Input: Pre-trained word vectors (e.g., GloVe, SG, CBOW, LBL) and a semantic lexicon graph defining relations (e.g., synonymy, hyponymy) between words.

Output: Updated word vectors after applying the belief propagation-based retrofitting objective to enforce semantic proximity.

Scoring recipe

def compute_metrics(predictions, gold):
    # For word similarity tasks (MEN-3k, RG-65, WS-353)
    spearman = scipy.stats.spearmanr(predictions, gold).correlation
    # For classification tasks (TOEFL, SYN-REL, SA)
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    return {'spearman': spearman, 'accuracy': acc}

Common pitfalls

• SYN-REL is inherently a syntactic task; incorporating semantic information via retrofitting can degrade performance on this benchmark.
• FrameNet lexicons may hurt performance because its frames group words based on abstract concepts rather than direct lexical relations.
• Multilingual benchmarks (RG-65, MC-30) are manually translated word-by-word from English, which may introduce noise or cultural bias.

Evidence (verbatim from paper)

Spearman’s correlation (3 left columns) and accuracy (3 right columns) on different tasks. Higher scores are always better.

Citation

@misc{faruqui2014retrofitting,
  title={Retrofitting Word Vectors to Semantic Lexicons},
  author={Faruqui et al. (2014)},
  year={2014},
  note={arXiv:1411.4166}
}

• arXiv: 1411.4166