Semeval2020 Semantic Change Eval

semeval2020-semantic-change-eval

Transparent Semantic Change Detection with Dependency-Based Profiles — Bach Phan-Tat et al. (2026) (arXiv:2601.02891, 2026)

What this evaluates

Evaluates a model's ability to detect and rank lexical semantic change over time across multiple languages. It probes both binary classification of whether a word's meaning has changed and graded ranking of the magnitude of that change.

Datasets

• SemEval 2020 Unsupervised Lexical Semantic Change Detection — total ?; splits: test (-1)

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly classified words. For Subtask 1, a word is labeled as changed if it falls in the top 43% of predicted JSD scores per language.
• Spearman's rank correlation (primary) — range: [-1, 1]
  • Pearson correlation coefficient between the ranks of predicted change scores (derived from JSD) and the gold change scores.

Input / output format

Input: Target words with associated slot-fillers and context across two time periods, along with gold change scores or labels.

Output: Subtask 2: A ranked list of target words by predicted change magnitude. Subtask 1: A binary label (changed/not changed) for each target word, derived by thresholding the top 43% of predicted scores per language.

Scoring recipe

# Subtask 2 (Spearman's rank correlation)
import scipy.stats as stats
spearman_corr = stats.spearmanr(gold_ranks, predicted_ranks).correlation

# Subtask 1 (Accuracy)
top_k = int(len(words) * 0.43)
predicted_labels = [1 if rank <= top_k else 0 for rank in predicted_ranks]
accuracy = sum(p == g for p, g in zip(predicted_labels, gold_labels)) / len(words)

Common pitfalls

• Using a fixed global threshold for Subtask 1 instead of the per-language top 43% cutoff specified in the shared task.
• Ignoring frequency filtering and POS tag removal, which the authors show substantially impact JSD scores and final rankings.
• Assuming dynamic programming is required for thresholding, as the paper notes a hard-coded 43% cutoff performs comparably.

Evidence (verbatim from paper)

Subtask 1 is a binary classification task and is evaluated based on accuracy. Subtask 2 is a ranking task and is evaluated with Spearman's rank correlation. The main focus of our method would be on subtask 2. We then use our subtask 2 scores for subtask 1 classification, following the strategy of Kutuzov et al. (2021).

Citation

@misc{bachphantat2026transparent,
  title={Transparent Semantic Change Detection with Dependency-Based Profiles},
  author={Bach Phan-Tat et al. (2026)},
  year={2026},
  note={arXiv:2601.02891}
}

• arXiv: 2601.02891