Semantic Syntactic Word Eval

semantic-syntactic-word-eval

Efficient Estimation of Word Representations in Vector Space — Mikolov et al. (2013) (arXiv:1301.3781, 2013)

What this evaluates

Evaluates whether trained word vector models can capture semantic and syntactic relationships between words through simple algebraic operations in vector space. It probes the model's ability to solve analogy-style questions by measuring how well the vector arithmetic preserves linguistic regularities.

Datasets

• Semantic-Syntactic Word Relationship test set — total 19544; splits: test (19544)

Metrics

• accuracy (primary) — range: percent
  • Percentage of analogy questions where the predicted word exactly matches the gold answer word. Computed as (number of correct predictions / total questions) * 100.

Input / output format

Input: An analogy question defined by two word pairs (A:B :: C:D), where the model receives the word vectors for A, B, and C.

Output: A single word token representing the predicted answer D.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred == gold:  # Exact match; synonyms count as mistakes
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Synonyms are explicitly counted as mistakes; only exact word matches are accepted.
• Multi-word entities (e.g., 'New York') are excluded from the test set, so models are only evaluated on single-token words.
• Reaching 100% accuracy is considered impossible due to the lack of morphological information in the models.

Evidence (verbatim from paper)

Question is assumed to be correctly answered only if the closest word to the vector computed using the above method is exactly the same as the correct word in the question; synonyms are thus counted as mistakes. This also means that reaching 100% accuracy is likely to be impossible, as the current models do not have any input information about word morphology.

Citation

@misc{mikolov2013word2vec,
  title={Efficient Estimation of Word Representations in Vector Space},
  author={Mikolov et al. (2013)},
  year={2013},
  note={arXiv:1301.3781}
}

• arXiv: 1301.3781