Yelp13 Sentiment Eval

yelp13-sentiment-eval

From Sentiment Annotations to Sentiment Prediction through Discourse Augmentation — Huber et al. (2020) (arXiv:2011.03021, 2020)

What this evaluates

This benchmark evaluates a model's ability to predict sentiment on long, complex documents by leveraging discourse structure. It probes whether incorporating hierarchical discourse trees improves sentiment classification and regression over standard sequential baselines, particularly for longer texts where sentiment is more subtle and diverse.

Datasets

• Yelp'13 — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1] | percent
  • Proportion of correctly predicted sentiment labels out of total instances.
• F1-score — range: [0, 1] | percent
  • Harmonic mean of precision and recall across the 5 sentiment classes, typically macro-averaged.
• Mean-Squared-Error (MSE) — range: [0, 4] | other
  • Average of the squared differences between predicted and actual sentiment scores on the 5-point scale.
• Mean-Absolute-Error (MAE) — range: [0, 4] | other
  • Average of the absolute differences between predicted and actual sentiment scores on the 5-point scale.

Input / output format

Input: Raw customer review documents, optionally pre-segmented into sentences or EDUs, with optional discourse tree structures (constituency/dependency) provided as input features.

Output: A predicted sentiment label on a 5-point ordinal scale (1 to 5), or a continuous regression score corresponding to the 5-point scale.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions and gold are lists of ints in [1, 5]
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    f1 = f1_score(gold, predictions, average='macro')
    mse = mean((np.array(predictions) - np.array(gold))**2)
    mae = mean(abs(np.array(predictions) - np.array(gold)))
    return acc, f1, mse, mae

Common pitfalls

• Data leakage from discourse parser training: The MEGA-DT corpus is derived from Yelp'13. Evaluators must ensure the discourse parser is only trained on the Yelp'13 training split, not the dev/test splits.
• Classification vs. Regression framing: The 5-point scale is ordinal. Treating it strictly as classification ignores the regression metrics (MSE/MAE) that capture ordinal distance, which the paper argues is crucial for rigorous evaluation.
• Ignoring document length stratification: Overall averages can mask performance differences. The paper shows discourse models significantly outperform baselines only on longer documents (>632 words).

Evidence (verbatim from paper)

To more rigorously evaluate the models in our evaluation, we show four metrics for each system, including the commonly used accuracy and F1-score, as well as the Mean-Squared-Error (MSE) and Mean-Absolute-Error (MAE) metrics.

Citation

@misc{huber2020sentiment,
  title={From Sentiment Annotations to Sentiment Prediction through Discourse Augmentation},
  author={Huber et al. (2020)},
  year={2020},
  note={arXiv:2011.03021}
}

• arXiv: 2011.03021