# Preposition Sense Disambiguation Eval

> Evaluates a model's ability to classify the sense of a preposition in context. It tests cross-lingual context representation and semi-supervised learning for fine-grained lexical disambiguation. Use when the user wants to benchmark on Web-reviews corpus, SemEval corpus, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/preposition-sense-disambiguation-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/preposition-sense-disambiguation-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/preposition-sense-disambiguation-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/preposition-sense-disambiguation-eval

---


# preposition-sense-disambiguation-eval

> Semi Supervised Preposition-Sense Disambiguation using Multilingual Data — Gonen et al. (2016) (arXiv:1611.08813, 2016)

## What this evaluates

Evaluates a model's ability to classify the sense of a preposition in context. It tests cross-lingual context representation and semi-supervised learning for fine-grained lexical disambiguation.

## Datasets

- **Web-reviews corpus** — total 4250; splits: train (2552), dev (845), test (853)
- **SemEval corpus** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified preposition senses out of the total number of test instances.

## Input / output format

**Input**: Context window surrounding a target preposition, typically represented as a sequence of words or features (e.g., POS tags, lemmas) fed into an LSTM context-encoder and MLP classifier.

**Output**: A predicted sense label for the target preposition from a predefined set of senses per preposition.

## Scoring recipe

```python
correct = 0
total = 0
for instance in test_set:
    pred = model.predict(instance.context)
    if pred == instance.gold_sense:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- The paper reports average accuracy over 5 different random seeds, not just a single run. Readers must replicate the 5-seed averaging to match reported numbers.
- The SemEval corpus allows multiple correct senses per instance; the loss function is adapted to sum probabilities over all correct classes, but accuracy is still computed as exact match against the gold label(s).

## Evidence (verbatim from paper)

> The accuracies we report are the average accuracies over 5 different seeds. We use the train/test split provided with the corpus. We further split the train set into train and dev sets, by assigning every fourth example of each sense to the dev set, yielding 2552/845/853 instances of train/dev/test. The model performs well also on the SemEval corpus, achieving an accuracy of 76.9.

## Citation

```bibtex
@misc{gonen2016semisupervised,
  title={Semi Supervised Preposition-Sense Disambiguation using Multilingual Data},
  author={Gonen et al. (2016)},
  year={2016},
  note={arXiv:1611.08813}
}
```

- arXiv: 1611.08813