Lscdiscovery Eval

lscdiscovery-eval

LSCDiscovery: A shared task on semantic change discovery and detection in Spanish — Zamora-Reina et al. (2022) (arXiv:2205.06691, 2022)

What this evaluates

Evaluates models' ability to detect and rank lexical semantic change in Spanish diachronic corpora. It probes both graded ranking of semantic shift magnitude and binary classification of sense gain/loss or change presence.

Datasets

• LSCDiscovery — total ?; splits: test (-1)

Metrics

• Spearman rank correlation (SPR) (primary) — range: [-1, 1]
  • Spearman's rank correlation coefficient ($\rho$) between predicted semantic change scores and gold graded scores. Ranges from -1 to 1, where 1 indicates perfect rank agreement.
• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for binary classification tasks (Binary Change Detection, Sense Gain, Sense Loss). Calculated as 2 * (P * R) / (P + R).

Input / output format

Input: Diachronic word usage pairs or sequences across time periods for a set of target words.

Output: Phase 1: A ranked list or continuous score for each target word indicating semantic shift magnitude. Phase 2: Binary labels (e.g., 0/1 for change/no change, or gain/loss) for each target word.

Scoring recipe

# Phase 1: Graded Change / COMPARE Discovery
def score_graded(pred_scores, gold_scores):
    return spearmanr(pred_scores, gold_scores).correlation

# Phase 2: Binary Change / Sense Gain / Sense Loss Detection
def score_binary(pred_labels, gold_labels):
    tp = sum(p == g == 1 for p, g in zip(pred_labels, gold_labels))
    fp = sum(p == 1 and g == 0 for p, g in zip(pred_labels, gold_labels))
    fn = sum(p == 0 and g == 1 for p, g in zip(pred_labels, gold_labels))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Top-performing systems often bypassed modeling the actual annotation procedure (e.g., JSD or COMPARE scores) and instead exploited strong correlations between different scoring metrics, limiting their generalizability.
• Binarizing graded predictions via fixed thresholds (e.g., percentiles) has a hard performance upper bound, as perfectly modeling the underlying continuous score still cannot achieve perfect binary classification.
• Type-based systems consistently underperformed token-based systems in this shared task, contrary to earlier benchmarks, due to biases in contextualized embeddings and aggregation methods.

Evidence (verbatim from paper)

Teams are ranked according to SPR score for the Graded Change subtask in decreasing order. The values corresponding to the three best systems are highlighted in bold type. ... Teams are ranked according to F1 score for subtask Change binary in decreasing order.

Citation

@misc{zamorareina2022lscdiscovery,
  title={LSCDiscovery: A shared task on semantic change discovery and detection in Spanish},
  author={Zamora-Reina et al. (2022)},
  year={2022},
  note={arXiv:2205.06691}
}

• arXiv: 2205.06691