Unlearning Recsys Eval

unlearning-recsys-eval

Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems — Lubitzsch et al. (2025) (arXiv:2508.17076, 2025)

What this evaluates

Evaluates the ability of recommender systems to efficiently remove specific user interactions or sensitive items (unlearning) while preserving recommendation utility. It probes real-world operational constraints, including handling sequential small-batch deletion requests, domain-specific triggers, and low-latency execution across collaborative filtering, session-based, and next-basket recommendation tasks.

Datasets

• TaFeng — total 817741; splits: unspecified (-1)
• Dunnhumby — total 2595732; splits: unspecified (-1)
• Instacart — total 3346083; splits: unspecified (-1); repo https://github.com/khanhnamle1994/instacart-orders/
• RSC15 — total 33003944; splits: unspecified (-1)
• DIGI — total 1235381; splits: unspecified (-1)
• NOWP — total 1291251677; splits: unspecified (-1)
• Goodreads — total 228648342; splits: unspecified (-1)
• MovieLens — total 20000263; splits: unspecified (-1)
• Amazon Reviews — total 571540000; splits: unspecified (-1); repo https://amazon-reviews-2023.github.io/

Metrics

• Recall, PHR, nDCG (primary) — range: [0, 1]
  • Standard ranking metrics. Recall@k measures the fraction of relevant items in the top-k recommendations. PHR is the probability that a relevant item appears in the top-k. nDCG@k discounts relevance by position using logarithmic discounts.
• Unlearning completeness — range: [0, 1] or distance
  • Empirical measure of how well the unlearned model approximates a model retrained without the forget set. Calculated via weight-space distance, output distribution divergence, or hypothesis testing for (ε,δ)-unlearning.
• Unlearning efficiency — range: seconds/memory
  • Runtime in seconds, memory usage, and hardware requirements to execute the unlearning request. May be split into offline precomputation time and online execution time.

Input / output format

Input: User interaction history (sequences or sets of items) and a forget set specifying interactions, users, or items to unlearn.

Output: Ranked list of recommended items for the next interaction or basket.

Scoring recipe

def evaluate(predictions, gold, forget_set, retrained_model):
    # Recommendation utility
    recall = sum(1 for p in predictions[:k] if p in gold) / len(gold)
    ndcg = compute_ndcg(predictions, gold)
    
    # Unlearning completeness
    unlearned_weights = get_model_weights()
    retrained_weights = retrained_model.get_weights()
    completeness = cosine_distance(unlearned_weights, retrained_weights)
    
    # Efficiency
    runtime = time_execution(unlearn)
    
    return {'recall': recall, 'nDCG': ndcg, 'completeness': completeness, 'runtime': runtime}

Common pitfalls

• Evaluating only single, large-batch unlearning requests instead of sequential small batches
• Ignoring latency and memory constraints in favor of pure accuracy
• Focusing exclusively on collaborative filtering while neglecting sequential/session-based tasks
• Using exact unlearning methods that scale poorly with dataset size

Evidence (verbatim from paper)

We propose to leverage common ranking metrics such as Recall, Personalized hit ratio (PHR) and Normalized Discounted Cumulative Gain (nDCG). ... We propose to measure the time it takes to execute unlearning requests, as well as the required memory and hardware (e.g., if accelerator hardware is needed).

Citation

@misc{lubitzsch2025unlearningrecsys,
  title={Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems},
  author={Lubitzsch et al. (2025)},
  year={2025},
  note={arXiv:2508.17076}
}

• arXiv: 2508.17076