News Rec Eval

news-rec-eval

A Survey on LLM-based News Recommender Systems — Wang et al. (2025) (arXiv:2502.09797, 2025)

What this evaluates

Evaluates the classification and ranking performance of LLM-based versus deep learning-based news recommendation models. It also measures the diversity of recommended items and how well the recommendations align with individual user history (personalization).

Datasets

• MIND-small — total ?; splits: (unstated)
• Adressa (one-week) — total ?; splits: (unstated)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the ROC Curve. Calculated as the fraction of positive-negative sample pairs where the predicted score for the positive sample exceeds that of the negative sample: AUC = (1/|P||N|) * sum_{p in P, n in N} I(s(p) > s(n)).
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. Average of the reciprocal of the rank position of the first relevant item across queries: MRR = (1/|Q|) * sum_{i=1}^{|Q|} (1/Rank_i).
• nDCG@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k. Ratio of the DCG of the predicted ranking to the ideal DCG: nDCG@k = sum_{i=1}^k (rel_i / log2(i+1)) / sum_{i=1}^k (rel*_i / log2(i+1)).
• Recall@k — range: [0, 1]
  • Fraction of actual positive samples correctly retrieved in the top-k list: Recall = TP / (TP + FN).
• Precision@k — range: [0, 1]
  • Fraction of retrieved items in the top-k list that are relevant: Precision = TP / (TP + FP).
• HR@k — range: [0, 1]
  • Hit Rate at k. Proportion of users for whom at least one relevant item appears in the top-k recommendations: HR@k = (1/|U|) * sum_{u in U} I(R_u intersect R_hat_u(k) != emptyset).
• Diversity@A_p@k — range: [0, 1]
  • Aspect-based diversity. Normalized entropy over aspect distributions in the top-k list: D_Ap@k = -sum_{j in A_p} (p(j) log p(j) / log(|A_p|)).
• Personalization@A_p@k — range: [0, 1]
  • Jaccard similarity between user history and recommendation list aspect probabilities: PS_Ap@k = sum_j min(R_j, H_j) / sum_j max(R_j, H_j).

Input / output format

Input: User interaction logs (IDs, clicks, impressions, demographics like region/time/browser) paired with news metadata (titles, abstracts, categories, entities, bodies).

Output: Predicted relevance scores for candidate news items, or a top-k ranked list of recommended news.

Scoring recipe

def compute_metrics(pred_scores, true_labels, k=5):
    ranked = np.argsort(pred_scores)[::-1][:k]
    hits = np.isin(ranked, np.where(true_labels)[0])
    hr = hits.mean()
    tp = hits.sum()
    precision = tp / k
    recall = tp / max(true_labels.sum(), 1)
    # MRR: 1/rank of first hit if any, else 0
    mrr = 1.0 / (np.where(hits)[0][0] + 1) if hits.any() else 0.0
    # nDCG@k: DCG / IDCG
    dcg = np.sum(hits / np.log2(np.arange(2, k+2)))
    idcg = np.sum(np.sort(true_labels)[:k][::-1] / np.log2(np.arange(2, k+2)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    # AUC: pair-wise comparison
    auc = np.mean([1 if pred_scores[p] > pred_scores[n] else 0 for p in np.where(true_labels)[0] for n in np.where(~true_labels)[0]])
    return {'AUC': auc, 'MRR': mrr, 'nDCG@k': ndcg, 'Recall@k': recall, 'Precision@k': precision, 'HR@k': hr}

Common pitfalls

• Only MIND-small and one-week Adressa subsets are used due to resource constraints, which may not reflect full-scale performance.
• Conflicting results on Adressa arise from LLMs' difficulty with multilingual news encoding compared to DL baselines, not necessarily model architecture flaws.
• Negative sampling strategy for AUC and tie-breaking rules for ranking metrics are not explicitly defined in the protocol.

Evidence (verbatim from paper)

In our experiments, we compute top 5 and 10 scores in terms of nDCG, Hit, Recall, Precision, diversity, and personalization.

Citation

@misc{wang2025surveyllmnewsrec,
  title={A Survey on LLM-based News Recommender Systems},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.09797}
}

• arXiv: 2502.09797