Dynamic Unlearning Eval

dynamic-unlearning-eval

The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning — Raj Sanjay Shah et al. (arXiv:2603.11266, 2026)

What this evaluates

Evaluates the effectiveness and robustness of LLM unlearning methods by measuring residual knowledge retrieval across dynamically generated single-hop, multi-hop, and alias-based queries, alongside the retention of adjacent and general knowledge.

Datasets

• RWKU — total ?; splits: test (-1)
• TOFU — total ?; splits: test (-1)

Metrics

• Multi-hop Forgetting Criterion (primary) — range: percent
  • Accuracy of residual knowledge retrieval on dynamically generated 1-hop, 2-hop, and 3-hop queries. Lower values indicate better unlearning performance.
• Retention Criterion — range: percent
  • Accuracy of retaining facts 1-fact away, 2-facts away, and relative retention of general knowledge. Higher values indicate better knowledge preservation.
• Overall Score — range: percent
  • Aggregated metric balancing forgetting and retention performance across all query types and proximity levels.

Input / output format

Input: Structured natural language queries generated dynamically from knowledge graphs, including single-hop, multi-hop (1- to 3-hop), and alias-perturbed variants, prompted to the LLM.

Output: Text completion answering the query. Evaluated for factual accuracy regarding the unlearned entity or retained adjacent facts.

Scoring recipe

for query in dynamic_benchmark:
  pred = model.generate(query)
  if is_forgetting_query(query):
    resolved = check_factual_accuracy(pred, gold_entity)
    forgetting_accs.append(resolved)
  else:
    retained = check_factual_accuracy(pred, gold_retained_fact)
    retention_accs.append(retained)
forget_score = mean(forgetting_accs) * 100
retain_score = mean(retention_accs) * 100
overall_score = aggregate(forget_score, retain_score)

Common pitfalls

• Assuming static benchmarks fully capture unlearning efficacy; this framework shows multi-hop and alias queries reveal significant residual knowledge missed by static probes.
• Overestimating the effectiveness of Chain-of-Thought decomposition; the paper finds it yields negligible improvement in preventing residual knowledge recovery.
• Ignoring proximity effects; retention accuracy drops for facts directly adjacent (1-hop) to the unlearned target but improves for more distant facts (2-hop).

Evidence (verbatim from paper)

Our metric shows the same relative efficacy of methods as previous unlearning evaluation methods. Results for RWKU are summarized in Table 2, showing each unlearning method’s performance across the multi-hop forgetting criterion and the retention criterion.

Citation

@misc{shah2026unlearningmirage,
  title={The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning},
  author={Raj Sanjay Shah et al.},
  year={2026},
  note={arXiv:2603.11266}
}

• arXiv: 2603.11266