Opinion Summarization Eval

opinion-summarization-eval

Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization — Nath et al. (2024) (arXiv:2402.15473, 2024)

What this evaluates

Evaluates a model's ability to synthesize multiple product reviews into a single, coherent opinion summary. It probes the model's capacity to capture diverse aspects, maintain factual accuracy, and adhere to domain-specific stylistic constraints without relying on surface-level lexical overlap.

Datasets

• Amazon — total ?; splits: test (-1)
• Oposum+ — total ?; splits: test (-1)
• Flipkart — total ?; splits: test (-1)

Metrics

• human evaluation (primary) — range: other
  • Human domain experts rate the quality of generated opinion summaries given a set of product reviews. The authors explicitly state that overlap-based metrics like ROUGE are inadequate for this task due to first-person bias and aspect coverage issues, making human assessment the primary evaluation method.

Input / output format

Input: A set of product reviews (typically 8-10 per product) from a specific e-commerce domain.

Output: A single opinion summary that synthesizes the opinions across the provided reviews.

Scoring recipe

def score_human_eval(predictions, gold, annotators):
    scores = []
    for pred in predictions:
        # Annotators (domain experts) review input + generated summary
        rating = annotators.rate_quality(pred)
        scores.append(rating)
    return mean(scores)

Common pitfalls

• Over-reliance on ROUGE or other overlap-based metrics, which the authors explicitly note fail to capture first-person bias and comprehensive aspect coverage in opinion summaries.
• Small benchmark scale (32-147 products total), which limits statistical power and may not generalize across broader e-commerce categories.
• Human evaluation relies on domain experts rather than crowd workers, which can introduce specific inductive biases or domain-specific rating thresholds.

Evidence (verbatim from paper)

Due to these shortcomings, in our analysis, we do not rely much on overlap-based evaluations, such as Rouge. Rather we rely on human evaluations.

Citation

@misc{nath2024leveraging,
  title={Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization},
  author={Nath et al. (2024)},
  year={2024},
  note={arXiv:2402.15473}
}

• arXiv: 2402.15473