Sentiment Analysis Eval

sentiment-analysis-eval

Improving Sentiment Analysis with Multi-task Learning of Negation — Barnes et al. (2019) (arXiv:1906.07610, 2019)

What this evaluates

Evaluates a model's ability to classify text sentiment into binary or fine-grained polarity categories. Specifically probes how well the model handles negation scope and polarity disentanglement through multi-task learning.

Datasets

• SST-binary — total ?; splits: test (-1)
• SST-fine — total ?; splits: test (-1)
• SemEval-binary — total ?; splits: test (-1)
• SemEval-fine — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly classified instances out of the total number of instances. Reported as mean accuracy and standard deviation over five independent runs.

Input / output format

Input: Text snippets (sentences or phrases) from the respective sentiment datasets.

Output: Sentiment polarity label (binary: positive/negative; fine-grained: very negative, negative, neutral, positive, very positive).

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = correct / total

Common pitfalls

• Comparing models with different parameter counts without controlling for capacity, which confounds architectural improvements with model size.
• Using different pooling strategies (e.g., mean vs. max pooling) that significantly impact performance independent of the core architecture.
• Ignoring statistical significance testing across multiple runs/tasks without applying a Bonferroni correction.

Evidence (verbatim from paper)

Table 2 shows the mean accuracy and standard deviation of single-task sentiment models (STL), multi-task models with SFU auxiliary negation data (MTL-SFU) and multi-task models with ConanDoyle-neg auxiliary negation data (MTL-CD) over five runs.

Citation

@misc{barnes2019improving,
  title={Improving Sentiment Analysis with Multi-task Learning of Negation},
  author={Barnes et al. (2019)},
  year={2019},
  note={arXiv:1906.07610}
}

• arXiv: 1906.07610