Sst Sentiment Analysis Eval

sst-sentiment-analysis-eval

Revisiting Paraphrase Question Generator using Pairwise Discriminator — Patro et al. (2019) (arXiv:1912.13149, 2019)

What this evaluates

Evaluates a model's ability to classify the sentiment of movie review phrases into five fine-grained categories, measuring classification accuracy and error rates. The benchmark probes hierarchical sentiment understanding at the phrase level rather than the full sentence level.

Datasets

• Stanford Sentiment Treebank (SST) — total 222000; splits: train (126000), val (30000), test (66000)

Metrics

• Error Rate (Fine-Grained) (primary) — range: [0, 1]
  • 1 - Accuracy; calculated as the number of misclassified phrases divided by the total number of phrases in the fine-grained 5-class setting ({Very Negative, Negative, Neutral, Positive, Very Positive}). Lower values indicate better performance.

Input / output format

Input: Movie review phrase or sentence text (string)

Output: Sentiment label from {Very Negative, Negative, Neutral, Positive, Very Positive}

Scoring recipe

def compute_error_rate(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    error_rate = 1 - (correct / len(gold_labels))
    return error_rate

Common pitfalls

• The evaluation is performed at the phrase level, not the full sentence level, which differs from standard SST sentence-level benchmarks and requires phrase-level annotations.
• Error rate is reported instead of accuracy, so readers must remember that lower values indicate better performance, contrary to most classification benchmarks.

Evidence (verbatim from paper)

| Model | Error Rate (Fine-Grained) |

Citation

@misc{patro2019revisiting,
  title={Revisiting Paraphrase Question Generator using Pairwise Discriminator},
  author={Patro et al. (2019)},
  year={2019},
  note={arXiv:1912.13149}
}

• arXiv: 1912.13149