Bars Recommender Eval

bars-recommender-eval

BARS: Towards Open Benchmarking for Recommender Systems — Zhu et al. (2022) (arXiv:2205.09626, 2022)

What this evaluates

Evaluates the reproducibility and standardization of evaluation protocols in recommender systems. It probes both candidate item matching (ranking) and click-through rate (CTR) prediction tasks using standardized data splits, hyperparameter configurations, and common industry metrics to ensure fair and comparable model performance.

Datasets

• Criteo — total ?; splits: test (-1)
• MovieLens — total ?; splits: test (-1)

Metrics

• Recall@K — range: [0, 1]
  • Measures the fraction of relevant items successfully retrieved in the top-K recommendations. The paper defers exact mathematical definitions to later sections but notes it is a standard metric for candidate item matching.
• HitRate@K — range: [0, 1]
  • A binary indicator that equals 1 if at least one relevant item appears in the top-K recommendations, and 0 otherwise. Used for candidate item matching tasks.
• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K, which evaluates ranking quality by applying logarithmic discounting to position-based relevance. The paper recommends it as a standard metric for matching tasks.
• logloss — range: [0, 1]
  • Logarithmic loss for binary CTR prediction, measuring the calibration of predicted probabilities against actual click/no-click outcomes. Recommended for CTR prediction tasks.
• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the model's ability to rank positive interactions higher than negatives. Recommended as a standard metric for CTR prediction tasks.

Input / output format

Input: User-item interaction logs, categorical features for CTR prediction, and candidate item lists for matching tasks. Data is preprocessed with specific feature embedding dimensions (e.g., d=16 or d=40) and rare feature filtering (min_counts=10).

Output: Predicted relevance scores or CTR probabilities for candidate items, ranked or scored for evaluation against ground-truth interactions.

Scoring recipe

def compute_metrics(preds, gold, k=20):
    top_k = np.argsort(preds)[-k:][::-1]
    hits = [1 if idx in gold else 0 for idx in top_k]
    recall = sum(hits) / len(gold) if gold else 0
    dcg = sum(h / np.log2(i + 2) for i, h in enumerate(hits))
    idcg = sum(h / np.log2(i + 2) for i, h in enumerate(sorted(hits, reverse=True)))
    ndcg = dcg / idcg if idcg > 0 else 0
    auc = np.mean(np.array(preds)[np.array(gold)] > np.array(preds)[~np.array(gold)])
    logloss = -np.mean(np.log(np.clip(preds[gold], 1e-7, 1)))
    return recall, ndcg, auc, logloss

Common pitfalls

• Using arbitrary or unpublished data splits without sharing random seeds or preprocessing scripts, making results irreproducible and incomparable.
• Comparing models using different evaluation metrics or K values across papers, preventing direct performance benchmarking.
• Running models with default hyperparameters instead of properly tuned configurations, leading to unfair baseline comparisons and inconsistent performance claims.

Evidence (verbatim from paper)

To standardize the metrics for open benchmarking, we recommend to use Recall@K, HitRate@K, and NDCG@K for candidate item matching tasks, and for CTR prediction tasks, we recommend to use logloss and AUC. We defer their definitions in later sections. These metrics are mostly used in the literature and usually adopted in practice. Our benchmarking work reports the results of these standard metrics to allow for easy reuse.

Citation

@misc{zhu2022bars,
  title={BARS: Towards Open Benchmarking for Recommender Systems},
  author={Zhu et al. (2022)},
  year={2022},
  note={arXiv:2205.09626}
}

• arXiv: 2205.09626