Onerec Think Eval

onerec-think-eval

OneRec-Think: In-Text Reasoning for Generative Recommendation — Liu et al. (2025) (arXiv:2510.11639, 2025)

What this evaluates

Evaluates a generative recommendation model's ability to perform sequential next-item prediction and in-text reasoning for user preference alignment. It probes the model's capacity to generate interpretable reasoning paths alongside item recommendations, and measures ranking accuracy on standard recommendation benchmarks.

Datasets

• Amazon Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Amazon Toys — total ?; splits: train (-1), val (-1), test (-1)
• Amazon Sports — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• R@K / NDCG@K (primary) — range: [0, 1]
  • Recall@K: 1 if the ground-truth item appears in the top-K predicted items, else 0. NDCG@K: Normalized Discounted Cumulative Gain at K, which weights the relevance of the ground-truth item by the logarithmic position in the ranked list and normalizes by the ideal DCG.

Input / output format

Input: Sequential user interaction history (item tokens) for next-item prediction.

Output: Generated sequence of item tokens, optionally accompanied by in-text reasoning steps.

Scoring recipe

def compute_metrics(preds, gold, k=5):
    recall = 1.0 if gold in preds[:k] else 0.0
    if gold in preds:
        rank = preds.index(gold) + 1
        dcg = 1.0 / math.log2(rank + 1)
        idcg = 1.0 / math.log2(2)
        ndcg = dcg / idcg
    else:
        ndcg = 0.0
    return recall, ndcg

Common pitfalls

• Exact dataset split sizes are not provided in the main text; only the Amazon review benchmark source is cited.
• K is fixed at 5 and 10; results are not reported for other cutoffs.
• Industrial A/B test metrics (e.g., App Stay Time) are evaluated separately from the academic benchmark metrics and should not be conflated.

Evidence (verbatim from paper)

We use three real-world recommendation datasets from the popular Amazon review benchmark: Beauty, Toys, and Sports. We compare OneRec-Think against two groups of competitive baselines: (1) Classic sequential methods like BERT4Rec, HGN, GRU4Rec, and SASRec; and (2) Generative Recommender Models, such as TIGER, HSTU, and ReaRec. Top-K Recall (R@K) and NDCG (N@K) with K=5 and 10 are used as metrics, following Rajput et al. ([2023]).

Citation

@misc{liu2025onerecthink,
  title={OneRec-Think: In-Text Reasoning for Generative Recommendation},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.11639}
}

• arXiv: 2510.11639