Orbit Eval

orbit-eval

ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests — He et al. (2025) (arXiv:2510.26095, 2025)

What this evaluates

Evaluates recommendation models on candidate item ranking across multiple public sequential recommendation datasets and a large-scale synthetic hidden test (ClueWeb-Reco) to assess generalization to unseen item pools and real-world browsing scenarios.

Datasets

• ML-1M — total ?; splits: train (-1), test (-1)
• Amazon Beauty — total ?; splits: train (-1), test (-1)
• Amazon Toys — total ?; splits: train (-1), test (-1)
• Amazon Sports — total ?; splits: train (-1), test (-1)
• Amazon Books — total ?; splits: train (-1), test (-1)
• ClueWeb-Reco — total ?; splits: test (-1)

Metrics

• Recall@10 (primary) — range: [0, 1]
  • Fraction of relevant items correctly retrieved in the top-10 recommendations. Recall@K = |Predicted ∩ Relevant| / |Relevant|.
• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. NDCG@K = DCG@K / IDCG@K, where DCG@K = Σ_{i=1}^K (rel_i / log2(i+1)).
• Recall@50 — range: [0, 1]
  • Fraction of relevant items correctly retrieved in the top-50 recommendations.
• NDCG@50 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 50.
• Recall@100 — range: [0, 1]
  • Fraction of relevant items correctly retrieved in the top-100 recommendations.
• NDCG@100 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 100.

Input / output format

Input: User interaction history (sequential clicks/purchases) and a candidate item pool.

Output: A ranked list of recommended items (top-K).

Scoring recipe

def compute_metrics(pred_list, relevant_set, k):
    top_k = pred_list[:k]
    recall = len(set(top_k) & relevant_set) / len(relevant_set)
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(top_k) if item in relevant_set)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(k, len(relevant_set))))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    return recall, ndcg

Common pitfalls

• The ClueWeb-Reco hidden test evaluates zero-shot generalization to a massive, unseen item pool, making direct comparison with models trained on the same distribution misleading.
• Metrics are reported at multiple cutoffs (10, 50, 100); focusing solely on NDCG@10 may overlook deep-list recall capabilities crucial for real-world recommendation.
• Content-based and LLM baselines leverage rich item metadata, whereas ID-based models rely purely on interaction sequences; evaluation must account for this architectural difference.

Evidence (verbatim from paper)

Table 4 presents Recall@10 and NDCG@10 results, with key observations below: (1) We witness a consistent performance gain in sequential-based ID-based models through their evolution from RNN-based architecture to transformer architecture due to the attention-based structure is better at discovering long-term behavior patterns and user interests.

Citation

@misc{he2025orbit,
  title={ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2510.26095}
}

• arXiv: 2510.26095