Regen Quality Eval

regen-quality-eval

Less is More: Benchmarking LLM Based Recommendation Agents — Chauhan et al. (2026) (arXiv:2601.20316, 2026)

What this evaluates

Evaluates whether increasing the length of a user's purchase history context improves recommendation quality for LLM-based agents. It probes the saturation point of personalization reasoning and the cost-efficiency trade-off of context length.

Datasets

• REGEN — total 50; splits: test (50)

Metrics

• quality scores (primary) — range: [0, 1]
  • A recommendation accuracy metric reported on a scale of approximately 0.16 to 0.23. The exact mathematical formulation is not detailed in the results section, but it measures the relevance of the agent's recommendations against ground truth user interactions.

Input / output format

Input: A list of user purchase history items, truncated or padded to specific context lengths (5, 10, 15, 25, or 50 items).

Output: A ranked list of recommended items.

Scoring recipe

def compute_quality(predictions, gold):
    # Calculate relevance metric between predicted items and ground truth
    score = relevance_metric(predictions, gold)
    return score

# Average across 50 users per context length
avg_quality = mean([compute_quality(preds, gold) for preds, gold in zip(predictions, golds)])

Common pitfalls

• Assuming longer context lengths inherently improve recommendation accuracy without empirical verification.
• Ignoring the significant token cost and latency overhead when using maximal context lengths.
• Overlooking the within-subject experimental design, which requires paired statistical tests rather than independent comparisons.

Evidence (verbatim from paper)

Table 2 presents our complete results across all models and context lengths. The key observation is that quality scores remain remarkably stable regardless of context length, ranging from 0.16 to 0.23 across all conditions.

Citation

@misc{chauhan2026less,
  title={Less is More: Benchmarking LLM Based Recommendation Agents},
  author={Chauhan et al. (2026)},
  year={2026},
  note={arXiv:2601.20316}
}

• arXiv: 2601.20316