Gerestaurant Eval

gerestaurant-eval

GERestaurant: A German Dataset of Annotated Restaurant Reviews for Aspect-Based Sentiment Analysis — Hellwig et al. (2024) (arXiv:2408.07955, 2024)

What this evaluates

Evaluates Aspect-Based Sentiment Analysis (ABSA) capabilities on German-language restaurant reviews. It probes models on four subtasks: identifying aspect categories, predicting sentiment polarities for aspects, extracting aspect-sentiment pairs, and end-to-end triplet extraction.

Datasets

• GERestaurant — total 3078; splits: train (-1), test (-1); repo https://github.com/NilsHellwig/GERestaurant

Metrics

• F1 Micro (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, computed by aggregating total true positives, false positives, and false negatives across all classes and instances before calculating the score.
• F1 Macro — range: [0, 1]
  • Harmonic mean of precision and recall, computed independently for each class and then averaged across all classes, giving equal weight to each category regardless of frequency.

Input / output format

Input: German-language restaurant review sentences sourced from TripAdvisor.

Output: Predictions for four ABSA subtasks: aspect category labels (ACD), aspect-sentiment pairs (ACSA), aspect-term-category-polarity triplets (E2E-ABSA), and target aspect sentiment labels (TASD).

Scoring recipe

def compute_f1(preds, golds, average='micro'):
    # preds, golds: lists of predicted and true labels/triplets
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g and p in golds)
    fn = sum(1 for p, g in zip(preds, golds) if p != g and g not in preds)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Implicit aspects are annotated as NULL or require special handling, which can lower recall if models only predict explicit terms.
• Macro F1 is significantly lower than Micro F1 due to class imbalance, particularly for the PRICE and GENERAL-IMPRESSION categories.
• Aspect category labels must be mapped to the paper's schema (e.g., SemEval's PRICES -> PRICE, RESTAURANT -> GENERAL-IMPRESSION) to match baseline results.

Evidence (verbatim from paper)

The performance achieved in the four ABSA tasks under consideration are presented in Table [6]. For predicting the five aspect classes (ACD task), gbert-large demonstrated the highest performance, achieving micro and macro F1 scores of 91.82 and 90.73, respectively, placing it approximately three percentage points ahead of gbert-base. Similarly, in the classification of aspects combined with their polarity (ACSA), the best performance was observed when employing gbert-large, which attained micro and macro F1 scores of 85.14 and 58.61, respectively.

Citation

@misc{hellwig2024gerestaurant,
  title={GERestaurant: A German Dataset of Annotated Restaurant Reviews for Aspect-Based Sentiment Analysis},
  author={Hellwig et al. (2024)},
  year={2024},
  note={arXiv:2408.07955}
}

• arXiv: 2408.07955