Wsi Semcor Eval

wsi-semcor-eval

In the LLM era, Word Sense Induction remains unsolved — Mosolova et al. (2026) (arXiv:2603.11686, 2026)

What this evaluates

Evaluates Word Sense Induction (WSI) by clustering contextualized word embeddings to predict sense assignments. It probes a model's ability to capture lexical polysemy and contextual meaning without supervised sense labels, using natural corpus distributions rather than artificially skewed benchmarks.

Datasets

• SemCor — total ?; splits: test (-1)

Metrics

• F-B^3 (primary) — range: percent
  • A clustering metric that computes the harmonic mean of precision and recall by optimally matching induced clusters to gold sense assignments. It is calculated as F-B^3 = 2 * (precision * recall) / (precision + recall), typically scaled to a percentage.

Input / output format

Input: Contextualized embeddings of target words in their surrounding context, extracted from pre-trained or fine-tuned language models (e.g., BERT, MirrorWiC).

Output: A partitioning of the contextualized embeddings into clusters, where each cluster represents an induced word sense.

Scoring recipe

def compute_fb3(predictions, gold):
    # predictions: list of cluster IDs per instance
    # gold: list of gold sense labels per instance
    # 1. Build confusion matrix between clusters and gold senses
    # 2. Find optimal one-to-one matching (e.g., Hungarian algorithm)
    # 3. Compute precision = sum(matches) / total_clusters
    # 4. Compute recall = sum(matches) / total_senses
    # 5. Return F1 = 2 * (prec * rec) / (prec + rec) * 100

Common pitfalls

• Relying on artificial datasets like SemEval that distort natural polysemy and frequency distributions.
• Assuming complex clustering algorithms will outperform the simple 'one cluster per lemma' (1cpl) baseline, which actually performs best for adjectives and nouns.
• Ignoring part-of-speech variations, as polysemy rates and optimal clustering strategies differ significantly across verbs, nouns, and adjectives.

Evidence (verbatim from paper)

Table 3: F-B $^3$ performance across PLMs and clustering algorithms for each POS, for all POS (All POS), and the average over POS weighted by POS proportion in SemCor ( $w$ Avg).

Citation

@misc{mosolova2026llmwsiremainsunsolved,
  title={In the LLM era, Word Sense Induction remains unsolved},
  author={Mosolova et al. (2026)},
  year={2026},
  note={arXiv:2603.11686}
}

• arXiv: 2603.11686