Sparse Word Vector Eval

sparse-word-vector-eval

Sparse Overcomplete Word Vector Representations — Faruqui et al. (2015) (arXiv:1506.02004, 2015)

What this evaluates

Evaluates the quality and interpretability of sparse overcomplete word vector representations by measuring their ability to capture lexical similarity and perform downstream text classification tasks compared to dense baseline vectors.

Datasets

• SimLex — total ?; splits: test (-1)
• Senti. — total ?; splits: test (-1)
• TREC — total ?; splits: test (-1)
• Sports — total ?; splits: test (-1)
• Comp. — total ?; splits: test (-1)
• Relig. — total ?; splits: test (-1)
• NP — total ?; splits: test (-1)

Metrics

• correlation — range: [-1, 1]
  • Pearson or Spearman correlation coefficient between predicted similarity scores and human judgment scores.
• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances using an ℓ2-regularized logistic regression model trained on the word vectors.

Input / output format

Input: Word vectors (initial dense representations or transformed sparse/binarized vectors) used as features for similarity computation or logistic regression classification.

Output: For similarity tasks: a scalar similarity score. For classification tasks: a predicted class label.

Scoring recipe

# For classification tasks:
model = LogisticRegression(penalty='l2')
model.fit(train_vectors, train_labels)
preds = model.predict(test_vectors)
accuracy = np.mean(preds == test_labels)

# For similarity tasks:
corr = np.corrcoef(predicted_scores, human_scores)[0, 1]

Common pitfalls

• WS-353 is used only for hyperparameter tuning, not for final evaluation reporting.
• Performance must be averaged across all seven tasks and multiple initial vector types (Glove, SG, GC, Multi) to match reported results.
• Classification uses an ℓ2-regularized logistic regression model, not a neural network or SVM.

Evidence (verbatim from paper)

The first of these is a word similarity task, where the score is correlation with human judgments, and the others are classification accuracies of an ℓ2-regularized logistic regression model trained using the word vectors.

Citation

@misc{faruqui2015sparse,
  title={Sparse Overcomplete Word Vector Representations},
  author={Faruqui et al. (2015)},
  year={2015},
  note={arXiv:1506.02004}
}

• arXiv: 1506.02004