Russe2018 Wsi Eval

russe2018-wsi-eval

RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language — Panchenko et al. (2018) (arXiv:1803.05795, 2018)

What this evaluates

Evaluates a model's ability to perform Word Sense Induction (WSI) by clustering contextual usages of target words into sense groups without prior sense labels. It probes the system's capability to handle morphological complexity and free word order in Russian across different sense granularities.

Datasets

• wiki-wiki — total ?; splits: test (-1); repo https://github.com/nlpub/russe-wsi-kit
• bts-rnc — total ?; splits: test (-1); repo https://github.com/nlpub/russe-wsi-kit
• active-dict — total ?; splits: test (-1); repo https://github.com/nlpub/russe-wsi-kit

Metrics

• ARI (primary) — range: [-1, 1]
  • Adjusted Rand Index measuring the similarity between the predicted sense clusters and the gold sense labels. Values range from -1 to 1, with 1 indicating perfect agreement between partitions.

Input / output format

Input: Contextual sentences or phrases containing a target word.

Output: A partitioning of the contexts into clusters, where each cluster corresponds to a distinct sense of the target word.

Scoring recipe

def compute_ari(predictions, gold, words):
    ari_scores = []
    for word in unique_words:
        pred_w = [p for p, w in zip(predictions, words) if w == word]
        gold_w = [g for g, w in zip(gold, words) if w == word]
        ari_scores.append(adjusted_rand_score(gold_w, pred_w))
    return sum(ari_scores) / len(ari_scores)

Common pitfalls

• Public and private test sets are non-overlapping; public covers ~1/3 of test words and is used for the leaderboard, while private determines final ranking.
• ARI scores vary drastically across datasets due to sense granularity; wiki-wiki contains coarse homonymous senses yielding high scores, while dictionary datasets contain fine-grained polysemous senses yielding lower scores.
• The task is unsupervised sense clustering; models must not rely on prior sense labels or external sense inventories.

Evidence (verbatim from paper)

Tables 2, 3, and 4 present the results of the shared task for the three datasets used for evaluation: wiki-wiki, bts-rnc, and active-dict. Each table lists top 10 best teams with the public and private ARI scores on the test set... Private and public scores were calculated on non-overlapping sets of words, with public words constituting approximately one third of all words in test set of each dataset.

Citation

@misc{panchenko2018russe,
  title={RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language},
  author={Panchenko et al. (2018)},
  year={2018},
  note={arXiv:1803.05795}
}

• arXiv: 1803.05795