Preposition Sense Disambiguation Eval

preposition-sense-disambiguation-eval

Semi Supervised Preposition-Sense Disambiguation using Multilingual Data — Gonen et al. (2016) (arXiv:1611.08813, 2016)

What this evaluates

Evaluates a model's ability to classify the sense of a preposition in context. It tests cross-lingual context representation and semi-supervised learning for fine-grained lexical disambiguation.

Datasets

• Web-reviews corpus — total 4250; splits: train (2552), dev (845), test (853)
• SemEval corpus — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified preposition senses out of the total number of test instances.

Input / output format

Input: Context window surrounding a target preposition, typically represented as a sequence of words or features (e.g., POS tags, lemmas) fed into an LSTM context-encoder and MLP classifier.

Output: A predicted sense label for the target preposition from a predefined set of senses per preposition.

Scoring recipe

correct = 0
total = 0
for instance in test_set:
    pred = model.predict(instance.context)
    if pred == instance.gold_sense:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• The paper reports average accuracy over 5 different random seeds, not just a single run. Readers must replicate the 5-seed averaging to match reported numbers.
• The SemEval corpus allows multiple correct senses per instance; the loss function is adapted to sum probabilities over all correct classes, but accuracy is still computed as exact match against the gold label(s).

Evidence (verbatim from paper)

The accuracies we report are the average accuracies over 5 different seeds. We use the train/test split provided with the corpus. We further split the train set into train and dev sets, by assigning every fourth example of each sense to the dev set, yielding 2552/845/853 instances of train/dev/test. The model performs well also on the SemEval corpus, achieving an accuracy of 76.9.

Citation

@misc{gonen2016semisupervised,
  title={Semi Supervised Preposition-Sense Disambiguation using Multilingual Data},
  author={Gonen et al. (2016)},
  year={2016},
  note={arXiv:1611.08813}
}

• arXiv: 1611.08813