Unsupervised Lexical Semantic Change Eval

unsupervised-lexical-semantic-change-eval

SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection — Schlechtweg et al. (2020) (arXiv:2007.11464, 2020)

What this evaluates

This benchmark evaluates a model's ability to detect unsupervised lexical semantic change across diachronic corpus pairs. It probes two capabilities: binary classification of whether a word's sense has been gained or lost, and ranking the intensity of semantic change relative to a gold standard.

Datasets

• SemEval-2020 Task 1 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted binary labels (sense gain/loss) out of total target words. Bounded between 0 and 1.
• Spearman’s rank-order correlation coefficient (primary) — range: [-1, 1]
  • Measures the monotonic relationship between predicted and gold rankings of change intensity. Ties are handled by assigning the average of the ranks that would have been assigned to all tied values. Bounded between -1 and 1.

Input / output format

Input: Diachronic corpus pairs for a target word in a specific language (English, German, Latin, or Swedish).

Output: Subtask 1: Binary label (0 for no change, 1 for change). Subtask 2: Predicted change intensity score or rank for each target word.

Scoring recipe

# Subtask 1: Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)

# Subtask 2: Spearman's rho
# 1. Rank predictions and gold labels (handle ties by average rank)
pred_ranks = rank_with_ties(predictions)
gold_ranks = rank_with_ties(gold_labels)
# 2. Compute correlation
rho = spearman_corr(pred_ranks, gold_ranks)

Common pitfalls

• Initial assumption of balanced class distribution for Subtask 1 proved incorrect for some languages, making accuracy less reliable than F1-score.
• Spearman’s ρ for Subtask 2 only considers the order of predicted values, not their actual magnitudes, so poorly calibrated scores can still yield high correlation.
• Ties in predicted change intensities must be resolved by assigning average ranks before computing correlation, otherwise the metric will be biased.

Evidence (verbatim from paper)

For Subtask 1 submitted predictions were evaluated against the hidden labels via accuracy, given that we anticipated the class distribution for target words to be approximately balanced before the annotation. Scores are bounded between 0 and 1. For Subtask 2, we used Spearman’s rank-order correlation coefficient ρ with the gold rank. Spearman’s ρ only considers the order of the words, the actual predicted change values were not taken into account.

Citation

@misc{schlechtweg2020semeval,
  title={SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection},
  author={Schlechtweg et al. (2020)},
  year={2020},
  note={arXiv:2007.11464}
}

• arXiv: 2007.11464