Diacr Ita Eval

diacr-ita-eval

CL-IMS @ DIACR-Ita: Volente o Nolente: BERT does not outperform SGNS on Semantic Change Detection — Laicher et al. (2020) (arXiv:2011.07247, 2020)

What this evaluates

Binary classification of lexical semantic change for target words across two diachronic time periods. It probes whether models can reliably detect meaning shifts in Italian using corpus pairs from newspapers and books.

Datasets

• DIACR-Ita — total 18; splits: test (18)

Metrics

• classification accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels (changed='1' or unchanged='0') out of the total number of target words.

Input / output format

Input: A list of target words and a pair of diachronic corpora ($C_1$ from time $t_1$, $C_2$ from time $t_2$).

Output: Binary labels ('1' for semantic change, '0' for no change) for each target word in the list.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Gold labels were withheld during the competition phase, preventing hyperparameter tuning on the test set.
• Teams could submit up to 4 predictions, and only the highest score per team was used for ranking, which may inflate reported performance if not standardized.

Evidence (verbatim from paper)

During the evaluation phase each team was allowed to submit up to 4 predictions for the full list of target words, which were scored using classification accuracy between the predicted labels and the gold data. The final competition ranking compares only the highest of the scores achieved by each team.

Citation

@misc{laicher2020clims,
  title={CL-IMS @ DIACR-Ita: Volente o Nolente: BERT does not outperform SGNS on Semantic Change Detection},
  author={Laicher et al. (2020)},
  year={2020},
  note={arXiv:2011.07247}
}

• arXiv: 2011.07247