Russe2020 Taxonomy Enrichment Eval

russe2020-taxonomy-enrichment-eval

RUSSE'2020: Findings of the First Taxonomy Enrichment Task for the Russian language — Nikishina et al. (2020) (arXiv:2005.11176, 2020)

What this evaluates

Evaluates a model's ability to extend an existing semantic hierarchy (RuWordNet) by predicting hypernym relationships for novel Russian words using only contextual corpus information, without relying on explicit word definitions. It probes contextual lexical grounding and unsupervised taxonomy extension capabilities specifically for Slavic languages.

Datasets

• RUSSE'2020 Taxonomy Enrichment Test Set — total ?; splits: test (-1)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision: computes the average precision at each rank where a relevant hypernym is retrieved, then averages across all test instances.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank: averages the reciprocal of the rank position of the first correctly predicted hypernym across all test instances.

Input / output format

Input: A novel Russian word (noun or verb) without a definition, accompanied by its contextual occurrences from text corpora.

Output: A ranked list of up to 10 candidate hypernym synsets from the existing RuWordNet taxonomy.

Scoring recipe

def compute_map_mrr(predictions, golds):
    aps, rrs = [], []
    for pred, gold_set in zip(predictions, golds):
        hits = 0
        ap = 0.0
        for i, p in enumerate(pred):
            if p in gold_set:
                hits += 1
                ap += hits / (i + 1)
        ap /= len(gold_set) if gold_set else 1
        aps.append(ap)
        rr = 1.0 / (pred.index(next(p for p in pred if p in gold_set)) + 1) if any(p in gold_set for p in pred) else 0.0
        rrs.append(rr)
    return sum(aps)/len(aps), sum(rrs)/len(rrs)

Common pitfalls

• Confusing this task with SemEval-2018 hypernym discovery, which requires building a taxonomy from scratch rather than extending an existing one.
• Assuming contextualized embeddings are necessary for top performance, as the best-performing systems actually relied on static word2vec vectors.
• Treating the test set as public; it is a private dataset derived from unpublished RuWordNet data.

Evidence (verbatim from paper)

If we compare the scores of SemEval participants and models submitted to our task, we can see that models participating in our task yielded significantly higher MRR scores — almost 0.6 for the best-performing models compared to 0.3 for the winners of SemEval.

Citation

@misc{nikishina2020russe,
  title={RUSSE'2020: Findings of the First Taxonomy Enrichment Task for the Russian language},
  author={Nikishina et al. (2020)},
  year={2020},
  note={arXiv:2005.11176}
}

• arXiv: 2005.11176