Mvn Text Classification Eval

mvn-text-classification-eval

End-to-End Multi-View Networks for Text Classification — Guo et al. (2017) (arXiv:1704.05907, 2017)

What this evaluates

Evaluates text classification models on sentiment analysis and news categorization tasks. It probes the model's ability to aggregate diverse feature views (word-level and n-gram) to predict fine-grained sentiment categories and news topics.

Datasets

• Stanford Sentiment Treebank — total 11855; splits: train (-1), dev (-1), test (-1)
• AG News — total 127600; splits: train (120000), test (7600)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted class labels out of the total number of test instances.
• error rate — range: [0, 1]
  • 1 - accuracy, reported for the AG News benchmark in Table 3.

Input / output format

Input: Raw text sentences (SST) or news articles (AG News), converted to 300-dimensional GloVe word embeddings and optionally augmented with CNN-derived n-gram features.

Output: Predicted class label (5 fine-grained sentiment classes for SST, 4 news categories for AG News).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# For AG News error rate:
error_rate = 1 - compute_accuracy(predictions, gold_labels)

Common pitfalls

• Evaluating on phrases instead of sentences for the Stanford Sentiment Treebank (paper explicitly restricts test evaluation to sentences only).
• Using custom or different train/dev/test splits instead of the exact splits from Socher et al. (2013).
• Confusing error rate with accuracy when reading AG News results, as Table 3 reports error rates while the text discusses accuracy improvements.

Evidence (verbatim from paper)

We use the same splits for training, dev, and test data as in (Socher et al., 2013) to predict the fine-grained 5-class sentiment categories of the sentences. ... The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table 1.

Citation

@misc{guo2017endtoend,
  title={End-to-End Multi-View Networks for Text Classification},
  author={Guo et al. (2017)},
  year={2017},
  note={arXiv:1704.05907}
}

• arXiv: 1704.05907