Sru Eval

sru-eval

On the Effectiveness of Unlearning in Session-Based Recommendation — Xin et al. (2023) (arXiv:2312.14447, 2023)

What this evaluates

Evaluates the recommendation accuracy and unlearning effectiveness of session-based recommendation models after deleting a portion of training sessions. It measures how well the model retains predictive performance while successfully preventing the inference of removed items.

Datasets

• Amazon Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Amazon Games — total ?; splits: train (-1), val (-1), test (-1)
• Steam — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K. Assigns higher scores to top-ranked positions. Computed as sum_{i=1}^K (rel_i / log2(i+1)) / ideal_NDCG@K.
• Recall@K — range: [0, 1]
  • Binary indicator: 1 if the ground-truth item appears in the top-K recommended items, 0 otherwise.
• HIT@K — range: [0, 1]
  • Probability that an unlearned item can be inferred or reconstructed from the remaining interactions. Lower scores indicate better unlearning effectiveness.

Input / output format

Input: Sequence of the last 10 interacted items (Beauty) or last 20 interacted items (Games, Steam), padded with a padding token if shorter.

Output: Ranked list of items from the entire item vocabulary.

Scoring recipe

def compute_metrics(preds, gold, k_vals=[10, 20]):
    scores = {}
    for k in k_vals:
        top_k = preds[:k]
        recall = 1.0 if gold in top_k else 0.0
        ndcg = 0.0
        for i, item in enumerate(top_k):
            if item == gold:
                ndcg = 1.0 / math.log2(i + 2)
                break
        scores[f'Recall@{k}'] = recall
        scores[f'NDCG@{k}'] = ndcg
    return scores

Common pitfalls

• HIT@K measures unlearning success, so lower scores indicate better performance (counter-intuitive for standard recommendation metrics).
• Exact unlearning is inherently impossible in session-based recommendation due to sequential and collaborative dependencies; metrics reflect residual inference probability rather than exact deletion.

Evidence (verbatim from paper)

To evaluate recommendation performance, we adopt two common top-K metrics: Recall@K and NDCG@K. Recall@K measures whether the ground-truth item is in the top-K positions of the recommendation list [38]. NDCG@K is a weighted metric that assigns higher scores to top-ranked positions [19]. We use the metric HIT described in section 4.4 to evaluate unlearning effectiveness.

Citation

@misc{xin2023sru,
  title={On the Effectiveness of Unlearning in Session-Based Recommendation},
  author={Xin et al. (2023)},
  year={2023},
  note={arXiv:2312.14447}
}

• arXiv: 2312.14447