Curriculum Word2vec Eval

curriculum-word2vec-eval

Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning — Tsvetkov et al. (2016) (arXiv:1605.03852, 2016)

What this evaluates

Evaluates how the ordering of training data (curriculum) affects the quality of task-specific word embeddings. It probes whether optimized data sequencing improves downstream performance in sentiment analysis, NER, POS tagging, and parsing compared to random or heuristic orderings.

Datasets

• Wikipedia Paragraph Corpus — total 2532361; splits: train (2532361), dev (-1), test (-1)

Metrics

• test results (primary) — range: percent
  • Standard classification accuracy (or F1 for some tasks) computed on the held-out test set for each of the four extrinsic tasks (Sentiment, NER, POS, Parse). Values are reported as percentages.

Input / output format

Input: Training: ordered Wikipedia paragraphs used to train 100-dim CBOW embeddings. Evaluation: downstream task instances (e.g., sentences for POS/NER/Parse/Senti) with word embeddings extracted from the trained model as features.

Output: Predicted label for each downstream task instance (e.g., POS tag, entity type, parse tree, sentiment class).

Scoring recipe

# 1. Train CBOW embeddings on ordered paragraphs (1 thread, 1 iteration)
embeddings = train_cbow(paragraphs, order=curriculum)
# 2. Extract embeddings as features for downstream task
features = [get_embedding(w) for w in sentence_tokens]
# 3. Train task classifier on dev set, predict on test set
model = train_classifier(features_dev, labels_dev)
preds = model.predict(features_test)
# 4. Compute accuracy
accuracy = sum(p == g for p, g in zip(preds, gold_labels)) / len(gold_labels) * 100

Common pitfalls

• Dev/test split incompatibility: best dev scores do not always correlate with best test scores, indicating potential distribution mismatch or small split sizes.
• Shuffled baselines require 10 independent training runs; the reported score uses the model closest to the median dev score, not the absolute best or average.
• Sequential processing is strictly enforced (single thread, 1 iteration) to isolate curriculum effects from parallel training speedups.

Evidence (verbatim from paper)

We tune the tasks on development data, and report results on the test data. The only component that varies across the experiments is order of paragraphs in the training corpus—the curriculum.

Citation

@misc{tsvetkov2016learning,
  title={Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning},
  author={Tsvetkov et al. (2016)},
  year={2016},
  note={arXiv:1605.03852}
}

• arXiv: 1605.03852