Imdb Sentiment Eval

imdb-sentiment-eval

Distributed Representations of Sentences and Documents — Quoc V. Le et al. (2014) (arXiv:1405.4053, 2014)

What this evaluates

Tests the model's capability to generate fixed-length representations for variable-length documents containing multiple sentences. It probes whether the method can scale to longer texts and outperform traditional bag-of-words baselines on a large-scale sentiment classification benchmark.

Datasets

• IMDB dataset — total 100000; splits: train (25000), test (25000), unlabeled_train (50000); repo http://ai.Stanford.edu/amaas/data/sentiment/index.html

Metrics

• error rate (primary) — range: percent
  • 1 - (number of correctly classified test instances / total test instances). Binary classification (Positive/Negative).

Input / output format

Input: Movie reviews consisting of multiple sentences.

Output: Predicted sentiment label (Positive or Negative).

Scoring recipe

predictions = classifier.predict(paragraph_vectors)
errors = sum(1 for p, g in zip(predictions, gold_labels) if p != g)
error_rate = errors / len(gold_labels)

Common pitfalls

• Paragraph vectors for test reviews are learned at test time via gradient descent after freezing the rest of the network.
• The model is trained on 75,000 documents (25k labeled + 50k unlabeled) to learn word and paragraph vectors before classification.
• A neural network with one hidden layer (50 units) is used instead of a linear logistic classifier.

Evidence (verbatim from paper)

The method described in this paper is the only approach that goes significantly beyond the barrier of 10% error rate. It achieves 7.42% which is another 1.3% absolute improvement (or 15% relative improvement) over the best previous result.

Citation

@misc{le2014distributed,
  title={Distributed Representations of Sentences and Documents},
  author={Quoc V. Le et al. (2014)},
  year={2014},
  note={arXiv:1405.4053}
}

• arXiv: 1405.4053