sst-sogou-news-eval
Molding CNNs for text: non-linear, non-consecutive convolutions — Lei et al. (2015) (arXiv:1508.04112, 2015)
What this evaluates
Evaluates text classification performance on sentence sentiment analysis and news topic categorization. Probes the model's ability to capture non-linear, non-consecutive word interactions (e.g., negation, long-range dependencies) for accurate document/sentence-level prediction.
Datasets
- Stanford Sentiment Treebank (Fine-grained) — total 11855; splits: train (8544), val (1101), test (2210)
- Stanford Sentiment Treebank (Binary) — total 9613; splits: train (6920), val (872), test (1821)
- Sogou Chinese News Corpora — total 99400; splits: train (79520), val (9940), test (9940)
Metrics
accuracy (primary) — range: percent
- Percentage of correctly predicted class labels out of the total number of instances in the evaluation split. Computed as (correct predictions / total instances) * 100.
Input / output format
Input: Sequence of pre-trained, fixed-dimension word vectors (300d GloVe for English, 200d word2vec for Chinese) representing sentences or documents. Embeddings are normalized to unit norm and kept fixed during training.
Output: Discrete class label (5-class fine-grained sentiment, binary sentiment, or 10-class news category).
Scoring recipe
def calculate_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Phrase-level annotations are added as separate training instances for sequence models, which can artificially inflate performance if not properly accounted for during evaluation.
- The binary sentiment split removes neutral sentences, drastically changing dataset size and class distribution compared to the fine-grained version.
- Word embeddings are fixed (not fine-tuned), which may underperform compared to models that update embeddings during training.
Evidence (verbatim from paper)
We evaluate our model on sentence sentiment classification task and news categorization task. For sentiment classification, we use the Stanford Sentiment Treebank benchmark... We use the standard 8544/1101/2210 split for training, development and testing respectively. Following previous work, we also evaluate our model on the binary classification variant of this benchmark, ignoring all neutral sentences. The binary version has 6920/872/1821 sentences for training, development and testing.
Citation
@misc{lei2015molding,
title={Molding CNNs for text: non-linear, non-consecutive convolutions},
author={Lei et al. (2015)},
year={2015},
note={arXiv:1508.04112}
}
1---2name: sst-sogou-news-eval3description: Evaluates text classification performance on sentence sentiment analysis and news topic categorization. Probes the model's ability to capture non-linear, non-consecutive word interactions (e.g., negation, long-range dependencies) for accurate document/sentence-level prediction. Use when the user wants to benchmark on Stanford Sentiment Treebank (Fine-grained), Stanford Sentiment Treebank (Binary), Sogou Chinese News Corpora, or asks about evaluating this task. Reports accuracy.4---56# sst-sogou-news-eval78> Molding CNNs for text: non-linear, non-consecutive convolutions — Lei et al. (2015) (arXiv:1508.04112, 2015)910## What this evaluates1112Evaluates text classification performance on sentence sentiment analysis and news topic categorization. Probes the model's ability to capture non-linear, non-consecutive word interactions (e.g., negation, long-range dependencies) for accurate document/sentence-level prediction.1314## Datasets1516- **Stanford Sentiment Treebank (Fine-grained)** — total 11855; splits: train (8544), val (1101), test (2210)17- **Stanford Sentiment Treebank (Binary)** — total 9613; splits: train (6920), val (872), test (1821)18- **Sogou Chinese News Corpora** — total 99400; splits: train (79520), val (9940), test (9940)1920## Metrics2122- `accuracy` **(primary)** — range: percent23 - Percentage of correctly predicted class labels out of the total number of instances in the evaluation split. Computed as (correct predictions / total instances) * 100.2425## Input / output format2627**Input**: Sequence of pre-trained, fixed-dimension word vectors (300d GloVe for English, 200d word2vec for Chinese) representing sentences or documents. Embeddings are normalized to unit norm and kept fixed during training.2829**Output**: Discrete class label (5-class fine-grained sentiment, binary sentiment, or 10-class news category).3031## Scoring recipe3233```python34def calculate_accuracy(predictions, gold_labels):35 correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)36 return (correct / len(gold_labels)) * 10037```3839## Common pitfalls4041- Phrase-level annotations are added as separate training instances for sequence models, which can artificially inflate performance if not properly accounted for during evaluation.42- The binary sentiment split removes neutral sentences, drastically changing dataset size and class distribution compared to the fine-grained version.43- Word embeddings are fixed (not fine-tuned), which may underperform compared to models that update embeddings during training.4445## Evidence (verbatim from paper)4647> We evaluate our model on sentence sentiment classification task and news categorization task. For sentiment classification, we use the Stanford Sentiment Treebank benchmark... We use the standard 8544/1101/2210 split for training, development and testing respectively. Following previous work, we also evaluate our model on the binary classification variant of this benchmark, ignoring all neutral sentences. The binary version has 6920/872/1821 sentences for training, development and testing.4849## Citation5051```bibtex52@misc{lei2015molding,53 title={Molding CNNs for text: non-linear, non-consecutive convolutions},54 author={Lei et al. (2015)},55 year={2015},56 note={arXiv:1508.04112}57}58```5960- arXiv: 1508.04112
qhjqhj00