Erase Eval

erase-eval

ERASE -- A Real-World Aligned Benchmark for Unlearning in Recommender Systems — Lubitzsch et al. (2026) (arXiv:2603.08341, 2026)

What this evaluates

Evaluates machine unlearning algorithms in recommender systems across collaborative filtering, session-based, and next-basket recommendation tasks. It measures how well models retain recommendation quality after unlearning sensitive or malicious user interactions, while maintaining computational efficiency and effectiveness compared to full retraining.

Datasets

• ERASE Benchmark (9 datasets) — total ?; splits: train (-1), test (-1); repo https://github.com/deem-data/erase-bench

Metrics

• utility (primary) — range: [0, 1]
  • Not specified in this section; generally measures the retention of recommendation performance (e.g., accuracy or ranking quality) after unlearning sensitive or malicious interactions.
• efficiency — range: other
  • Not specified in this section; refers to the runtime of the unlearning step. Real-world targets require unlearning to be at least three orders of magnitude faster than retraining from scratch.
• effectiveness — range: [0, 1]
  • Not specified in this section; measures the degree to which the unlearning algorithm successfully removes the specified interactions or user data from the model's predictions.

Input / output format

Input: Not specified in provided section. Typically involves a task-specific model trained on a dataset, a defined sequence of unlearning requests (forget set) based on a scenario, and an unlearning algorithm.

Output: Not specified in provided section. Models produce recommendations, which are then evaluated for utility, efficiency, and effectiveness against a baseline model retrained on the retain set.

Scoring recipe

# Pseudo-code for ERASE evaluation protocol
model = train(task_model, dataset)
forget_set = define_forget_set(scenario)  # e.g., sensitive items or spam users
unlearned_model = apply_unlearning(model, forget_set, algorithm)
utility = evaluate_recommendation_quality(unlearned_model, test_set)
efficiency = measure_runtime(unlearning_step)
effectiveness = check_forget_compliance(unlearned_model, forget_set)
baseline_model = train(task_model, dataset - forget_set)
compare(utility, efficiency, effectiveness, baseline_model)

Common pitfalls

• Existing benchmarks often use unrealistically large single unlearning requests (e.g., 5% of data) instead of sequential small updates that arrive over time.
• Runtime requirements are strict: unlearning should be at least three orders of magnitude faster than retraining, which many approximate methods fail to meet.
• General-purpose unlearning methods often fail under sequential unlearning, especially in attention and recurrent architectures, while recommender-specific methods show superior robustness.

Evidence (verbatim from paper)

Finally, the unlearned model is evaluated for utility, efficiency, and effectiveness, in comparison to a model retrained on the retain set. In practice, since models are retrained periodically anyway (e.g., weekly or monthly), unlearning is most useful for efficiently removing small batches of data between retraining cycles.

Citation

@misc{lubitzsch2026erase,
  title={ERASE -- A Real-World Aligned Benchmark for Unlearning in Recommender Systems},
  author={Lubitzsch et al. (2026)},
  year={2026},
  note={arXiv:2603.08341}
}

• arXiv: 2603.08341