Roast Review Level Absa Eval

roast-review-level-absa-eval

ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA — Chebolu et al. (2024) (arXiv:2405.20274, 2024)

What this evaluates

Evaluates a model's ability to jointly detect aspects, sentiments, targets, and opinions across entire product/course reviews, capturing contextual dependencies that sentence-level methods miss. It probes review-level joint extraction for both triplet (aspect-sentiment-target) and quadruple (aspect-sentiment-target-opinion) formats.

Datasets

• ROAST Benchmark (Amazon_FF, Coursera, Hotels, Phones, Movies) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RiTUAL-UH/ROAST-ABSA

Metrics

• F1 score (primary) — range: [0, 1]
  • Exact-match F1 score calculated over all predicted triplets and quadruples. A prediction is correct only if all corresponding elements (aspect, sentiment, target, opinion) exactly match the gold standard labels; partial matches are explicitly counted as incorrect.

Input / output format

Input: Raw review text (spanning multiple sentences) in English, Hindi, or Telugu.

Output: A list of predicted triplets or quadruples, where each contains the extracted aspect, sentiment, target, and opinion spans/labels.

Scoring recipe

def compute_exact_match_f1(preds, gold):
    correct = sum(1 for p in preds if p in gold)
    prec = correct / len(preds) if preds else 0
    rec = correct / len(gold) if gold else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Partial matches are explicitly treated as incorrect, so models must predict exact spans and labels for all elements simultaneously.
• Evaluation is performed at the review level, not sentence level, requiring models to handle cross-sentence dependencies and implicit elements.
• Results are averaged over 5 runs with different random seeds, so single-run reports may not match paper results.

Evidence (verbatim from paper)

Following Chebolu et al. ([2024]); Zhang et al. ([2021a]) experiments, we use the F1 score to measure the performance of different approaches on all the tasks. All experimental results are reported using the average of 5 different runs using distinct random seeds. We divided each domain dataset into train, validation, and test sets with 70%, 10%, and 20% splits, respectively. A triplet and quadruple is considered correct only if all the corresponding prediction elements match the gold standard labels. We consider any partial matches as wrong predictions following Zhang et al. ([2021a]).

Citation

@misc{chebolu2024roast,
  title={ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA},
  author={Chebolu et al. (2024)},
  year={2024},
  note={arXiv:2405.20274}
}

• arXiv: 2405.20274