# Opinion Summarization Eval

> Evaluates a model's ability to synthesize multiple product reviews into a single, coherent opinion summary. It probes the model's capacity to capture diverse aspects, maintain factual accuracy, and adhere to domain-specific stylistic constraints without relying on surface-level lexical overlap. Use when the user wants to benchmark on Amazon, Oposum+, Flipkart, or asks about evaluating this task. Reports human evaluation.

- Skill: `qhjqhj00/opinion-summarization-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/opinion-summarization-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/opinion-summarization-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/opinion-summarization-eval

---


# opinion-summarization-eval

> Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization — Nath et al. (2024) (arXiv:2402.15473, 2024)

## What this evaluates

Evaluates a model's ability to synthesize multiple product reviews into a single, coherent opinion summary. It probes the model's capacity to capture diverse aspects, maintain factual accuracy, and adhere to domain-specific stylistic constraints without relying on surface-level lexical overlap.

## Datasets

- **Amazon** — total ?; splits: test (-1)
- **Oposum+** — total ?; splits: test (-1)
- **Flipkart** — total ?; splits: test (-1)

## Metrics

- `human evaluation` **(primary)** — range: other
  - Human domain experts rate the quality of generated opinion summaries given a set of product reviews. The authors explicitly state that overlap-based metrics like ROUGE are inadequate for this task due to first-person bias and aspect coverage issues, making human assessment the primary evaluation method.

## Input / output format

**Input**: A set of product reviews (typically 8-10 per product) from a specific e-commerce domain.

**Output**: A single opinion summary that synthesizes the opinions across the provided reviews.

## Scoring recipe

```python
def score_human_eval(predictions, gold, annotators):
    scores = []
    for pred in predictions:
        # Annotators (domain experts) review input + generated summary
        rating = annotators.rate_quality(pred)
        scores.append(rating)
    return mean(scores)
```

## Common pitfalls

- Over-reliance on ROUGE or other overlap-based metrics, which the authors explicitly note fail to capture first-person bias and comprehensive aspect coverage in opinion summaries.
- Small benchmark scale (32-147 products total), which limits statistical power and may not generalize across broader e-commerce categories.
- Human evaluation relies on domain experts rather than crowd workers, which can introduce specific inductive biases or domain-specific rating thresholds.

## Evidence (verbatim from paper)

> Due to these shortcomings, in our analysis, we do not rely much on overlap-based evaluations, such as Rouge. Rather we rely on human evaluations.

## Citation

```bibtex
@misc{nath2024leveraging,
  title={Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization},
  author={Nath et al. (2024)},
  year={2024},
  note={arXiv:2402.15473}
}
```

- arXiv: 2402.15473