qhjqhj00 sst-sentiment-eval
Bidirectional Tree-Structured LSTM with Head Lexicalization — Teng et al. (2016) (arXiv:1611.06788, 2016)
What this evaluates
Evaluates a model's ability to perform sentiment classification on constituent trees, testing both fine-grained (5-class) and binary sentiment prediction at the sentence root and phrase levels.
Datasets
- Stanford Sentiment Treebank — total ?; splits: train (-1), test (-1)
- TREC — total ?; splits: train (5452), test (500), dev (500)
Metrics
accuracy(primary) — range: percent- Percentage of correctly classified instances. Evaluated separately for sentence root labels and phrase labels on SST, and for question type on TREC.
Input / output format
Input: Constituent parse tree with word tokens at leaves (SST), or raw question sentence (TREC).
Output: Sentiment label (0-4 for fine-grained, binary) for each node, or question type class (ENTY, HUM, LOC, DESC, NUM, ABBR).
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return correct / len(gold) * 100
Common pitfalls
- Evaluating only root-level accuracy ignores the phrase-level supervision used in training and testing for SST.
- TREC lacks a development set; the paper randomly splits 500 training examples for dev, which may introduce variance compared to standard fixed splits.
Evidence (verbatim from paper)
accuracy increases from 51.2 to 52.8, and the binary sentiment classification accuracy increases from 88.5 to 89.2, which demonstrates the effectiveness of the head lexicalization mechanism.
Citation
@misc{teng2016bidirectional,
title={Bidirectional Tree-Structured LSTM with Head Lexicalization},
author={Teng et al. (2016)},
year={2016},
note={arXiv:1611.06788}
}
- arXiv: 1611.06788