Wsd Rp Accuracy Eval

wsd-rp-accuracy-eval

Resolving Regular Polysemy in Named Entities — Shu-Kai Hsieh et al. (2024) (arXiv:2401.09758, 2024)

What this evaluates

Evaluates a model's ability to disambiguate word senses for both common nouns and proper nouns exhibiting regular polysemy. It probes contextual understanding and the capacity to leverage structured sense glosses and dot-object type classes to select the correct meaning from a candidate inventory.

Datasets

• WSD dataset (CWN 2.0) — total 45784; splits: train (36622), test (9162)
• RP dataset (Revised Mandarin Chinese Dictionary) — total 4507; splits: train (3641), test (866)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of test sentences where the model's highest-probability predicted sense matches the ground-truth sense.

Input / output format

Input: Context-gloss pairs formatted as [CLS] TEST-SENT [SEP] TGT, SENSE-DEF, SENSE-EX-SENT [SEP] for WSD, or [CLS] TEST-SENT [SEP] TGT, RPCLASS, RPCLASS-GLOSS [SEP] for RP. Target words are enclosed in angular brackets < >.

Output: Probability distribution over candidate senses; the model selects the sense with the highest probability as the prediction.

Scoring recipe

correct = 0
total = 0
for sentence in test_set:
    candidates = get_candidates(sentence.target_word)
    probs = model.predict(sentence.context, candidates)
    predicted_sense = candidates[probs.argmax()]
    if predicted_sense == sentence.gold_sense:
        correct += 1
    total += 1
accuracy = correct / total

Common pitfalls

• Evaluation is reported per example (sentence), but training batches use flattened context-gloss sequences, which can cause confusion about batch size vs. example count.
• MFS baseline accuracy varies drastically by subset (0% for simple words, 38% for complex) due to lack of sense distribution skew in the simple subset, which can mislead readers expecting standard corpus statistics.
• POS-guided filtering is a pre-processing step to reduce candidates, not the final metric; the model still predicts among filtered candidates.

Evidence (verbatim from paper)

We split the WSD dataset into WSD_train, WSD_test; RP dataset into RP_train, RP_test. The model’s prediction accuracies under different conditions are shown in Figure 2. Overall, the model achieves .86 accuracy in the WSD task and .88 in the RP task respectively.

Citation

@misc{hsieh2024resolving,
  title={Resolving Regular Polysemy in Named Entities},
  author={Shu-Kai Hsieh et al. (2024)},
  year={2024},
  note={arXiv:2401.09758}
}

• arXiv: 2401.09758