Horizon Eval

horizon-eval

HORIZON: A Benchmark for In-the-wild User Behaviour Modeling — Goel et al. (2026) (arXiv:2604.17259, 2026)

What this evaluates

Evaluates user behavior modeling capabilities across temporal generalization, cross-domain prediction, and unseen-user scenarios. It probes how well recommendation models and LLMs can generalize to out-of-distribution users and future time periods using real-world interaction sequences.

Datasets

• Amazon Reviews (HORIZON Benchmark) — total 54000000; splits: IND pool (53000000), OOD set (1000000), Training subset (100000), IND test (25000), OOD test (25000)

Metrics

• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank K. Sum of relevance scores discounted by log2(rank+1), normalized by ideal DCG.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. Average of 1/rank of the first relevant item across queries.
• Recall@K — range: [0, 1]
  • Fraction of relevant items retrieved in the top K recommendations.
• Precision@K — range: [0, 1]
  • Fraction of retrieved items in the top K that are relevant.

Input / output format

Input: User interaction history sequences (item IDs with timestamps) for recommendation tasks; for LLM tasks, standardized prompts encoding user behavior and item descriptions/IDs.

Output: Ranked list of recommended item IDs (top K) for recommendation tasks; retrieved item IDs for LLM retrieval tasks.

Scoring recipe

def compute_metrics(predictions, ground_truth, k_values=[10, 50, 100]):
    scores = {}
    for k in k_values:
        top_k = predictions[:k]
        scores[f'Recall@{k}'] = 1 if any(item in ground_truth for item in top_k) else 0
        scores[f'Precision@{k}'] = len(set(top_k) & set(ground_truth)) / k
        dcg = sum(1 / math.log2(i + 2) for i, item in enumerate(top_k) if item in ground_truth)
        idcg = sum(1 / math.log2(i + 2) for i in range(min(len(ground_truth), k)))
        scores[f'NDCG@{k}'] = dcg / idcg if idcg > 0 else 0
    for i, item in enumerate(predictions):
        if item in ground_truth:
            scores['MRR'] = 1 / (i + 1)
            break
    return scores

Common pitfalls

• Temporal cutoff τ=2020 is strictly enforced; using standard random or leave-one-out splits violates the benchmark's temporal generalization protocol.
• OOD and IND test sets are disjoint; mixing them or using the same pool for fine-tuning and evaluation causes data leakage and invalidates zero-shot/fine-tuning comparisons.
• Metrics are reported at K=10, 50, 100; reporting only K=10 or using ratio-based/leave-one-out evaluation strategies yields non-comparable results.

Evidence (verbatim from paper)

All models are trained with standardized hyperparameters and evaluated on our four evaluation settings using MRR, Recall@K, and NDCG@K for K=10,50,100. As we do not perform ranking across queries, we compute standard retrieval metrics i.e. Recall@K and Precision@K for K=10,50,100 to assess the effectiveness of the generated outputs in retrieving relevant items.

Citation

@misc{goel2026horizon,
  title={HORIZON: A Benchmark for In-the-wild User Behaviour Modeling},
  author={Goel et al. (2026)},
  year={2026},
  note={arXiv:2604.17259}
}

• arXiv: 2604.17259