Oats Eval

oats-eval

OATS: Opinion Aspect Target Sentiment Quadruple Extraction Dataset for Aspect-Based Sentiment Analysis — Chebolu et al. (2023) (arXiv:2309.13297, 2023)

What this evaluates

Evaluates joint extraction of aspect-based sentiment analysis elements (target, aspect, opinion, sentiment) at both sentence and review levels across multiple domains. Probes a model's ability to perform fine-grained, multi-element sentiment extraction and handle inter-sentence sentiment dynamics.

Datasets

• OATS — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RiTUAL-UH/OATS-ABSA

Metrics

• F1 (primary) — range: [0, 1]
  • Exact-match F1 score over extracted tuples, triplets, or quadruples. A prediction is correct only if all corresponding elements exactly match the gold standard; partial matches are counted as incorrect.

Input / output format

Input: Review or sentence text containing user opinions.

Output: Extracted tuples (aspect category, sentiment), triplets (aspect, opinion, sentiment), or quadruples (target, aspect, opinion, sentiment).

Scoring recipe

def compute_f1(predictions, gold):
    correct = sum(1 for p in predictions if p in gold)
    precision = correct / len(predictions) if predictions else 0
    recall = correct / len(gold) if gold else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Partial matches (e.g., correct aspect but wrong sentiment) are explicitly counted as incorrect, not partial credit.
• Results are averaged over 5 runs with different random seeds, not reported from a single run.
• The 80/10/10 split is applied per domain, not globally across the entire dataset.

Evidence (verbatim from paper)

Following Zhang et al. (2021a); Xu et al. (2023); Wan et al. (2020), we use the F1 score to measure the performance of different approaches on all the tasks from Section 4.1. All experimental results are reported using the average of 5 different runs using distinct random seeds. We divided each domain dataset into train, validation, and test sets with 80%, 10%, and 10% splits, respectively. A tuple, triplet, and quadruple is considered correct only if all the corresponding prediction elements match the gold standard labels. We consider any partial matches as wrong predictions following Zhang et al. (2021a).

Citation

@misc{chebolu2023oats,
  title={OATS: Opinion Aspect Target Sentiment Quadruple Extraction Dataset for Aspect-Based Sentiment Analysis},
  author={Chebolu et al. (2023)},
  year={2023},
  note={arXiv:2309.13297}
}

• arXiv: 2309.13297