Train O Matic Wsd Eval

train-o-matic-wsd-eval

Huge Automatically Extracted Training Sets for Multilingual Word Sense Disambiguation — Pasini et al. (2018) (arXiv:1805.04685, 2018)

What this evaluates

Evaluates the quality of automatically generated multilingual word sense disambiguation (WSD) training corpora by training a supervised WSD system (IMS) on them and measuring performance on standard WSD benchmark datasets. It probes whether synthetic sense-annotated data can match or exceed manually annotated corpora, particularly for low-resource languages.

Datasets

• Senseval-2, Senseval-3, SemEval-2007, SemEval-2013, SemEval-2015 — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed over predicted vs. gold word sense IDs per instance using the unified WSD evaluation framework.

Input / output format

Input: Target word, surrounding context sentence, and candidate word senses from a lexical resource (WordNet or BabelNet).

Output: Predicted sense ID for the target word.

Scoring recipe

def compute_f1(pred_senses, gold_senses):
    tp = sum(1 for p, g in zip(pred_senses, gold_senses) if p == g)
    fp = sum(1 for p, g in zip(pred_senses, gold_senses) if p != g)
    fn = sum(1 for p, g in zip(pred_senses, gold_senses) if p == g and g != 'no-sense')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return f1 * 100

Common pitfalls

• Parameter tuning (K=500, z=2.0) is performed on a small in-house development set rather than a held-out test set, which may inflate reported F1 scores.
• Comparisons against the MFS (Most Frequent Sense) baseline are included, which can mask improvements on rare or context-dependent senses.
• Different lexical resources (WordNet vs. BabelNet) and sense inventories are used across languages, making cross-lingual F1 scores not directly comparable.

Evidence (verbatim from paper)

The evaluation has been performed using the unified evaluation framework for Word Sense Disambiguation made available by Raganato et al. (2017), thus considering the following WSD shared tasks: Senseval-2 (Edmonds and Cotton, 2001), Senseval-3 (Snyder and Palmer, 2004), SemEval-2007 (Navigli et al., 2007), SemEval-2013 (Navigli et al., 2013) and SemEval-2015 (Moro and Navigli, 2015). Table 4: F1 of IMS trained on Train-o-Matic, OMSTI and SemCor, and MFS for the Senseval-2, Senseval-3, SemEval-07, SemEval-13 and SemEval-15 datasets.

Citation

@misc{pasini2018huge,
  title={Huge Automatically Extracted Training Sets for Multilingual Word Sense Disambiguation},
  author={Pasini et al. (2018)},
  year={2018},
  note={arXiv:1805.04685}
}

• arXiv: 1805.04685