Semantic Change Detection Eval

semantic-change-detection-eval

Contextualized language models for semantic change detection: lessons learned — Kutuzov et al. (2022) (arXiv:2209.00154, 2022)

What this evaluates

Evaluates the ability of contextualized language models to detect diachronic semantic change in words across different time periods. It probes whether models can accurately rank words by their degree of meaning shift compared to human-annotated gold standards.

Datasets

• SemEval-2020 Task 1 — total ?; splits: test (-1)
• GEMS — total ?; splits: test (-1)

Metrics

• Spearman's ρ (primary) — range: [-1, 1]
  • Spearman's rank correlation coefficient between the model's predicted semantic change scores and the human-annotated gold standard scores.

Input / output format

Input: Two time periods $t_1$ and $t_2$ with corresponding corpora $C_1$ and $C_2$, and a set of target words. A language model generates contextualized token embeddings for each occurrence of the target words in both corpora.

Output: A change score for each target word, computed via PRT, APD, or their ensemble, followed by ranking words by the magnitude of the score.

Scoring recipe

# Compute PRT score
proto1 = mean(embeddings_w_t1)
proto2 = mean(embeddings_w_t2)
prt_score = 1.0 / cosine_similarity(proto1, proto2)

# Compute APD score
apd_score = mean(cosine_distance(x, y) for x in embeddings_w_t1 for y in embeddings_w_t2)

# Ensemble score
final_score = (prt_score + apd_score) / 2.0

# Evaluate
spearman_rho = spearmanr(final_score, gold_standard_scores)

Common pitfalls

• Assuming a single method (PRT or APD) is universally optimal; performance depends on the dataset's gold score distribution.
• Confusing usage variance or syntactic shifts with actual semantic change, leading to false positives for words with stable meanings.

Evidence (verbatim from paper)

A change score is computed from these matrices, indicating the degree of semantic change undergone by a word between $t_1$ and $t_2$. The target words are ranked by this value. ... Table 1: Spearman correlation with the gold standard per test set for the best methods from (Kutuzov and Giulianielli, 2020) and our PRT/APD ensemble approach.

Citation

@misc{kutuzov2022contextualized,
  title={Contextualized language models for semantic change detection: lessons learned},
  author={Kutuzov et al. (2022)},
  year={2022},
  note={arXiv:2209.00154}
}

• arXiv: 2209.00154