Pebench Eval

pebench-eval

PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models — Zhaopan Xu et al. (arXiv:2503.12545, 2025)

What this evaluates

Evaluates multimodal large language models on their ability to selectively forget specific person or event concepts while preserving general knowledge. It probes cross-concept interference, unlearning efficacy, and the trade-off between forgetting targeted data and maintaining model utility.

Datasets

• PEBench — total 8000; splits: forget (-1), retain (-1)

Metrics

• Efficacy (primary) — range: [0, 1]
  • Precision measuring the proportion of correctly unlearned target samples (person or event). Higher values indicate successful forgetting.
• Generality — range: [0, 1]
  • Precision measuring unlearning success on related or unseen samples of the target concept, testing generalization of the forgetting process.
• Retain — range: [0, 1]
  • Precision measuring the preservation of non-target knowledge on the retain set after unlearning.
• Scope — range: [0, 1]
  • ROUGE-L score measuring the preservation of related descriptive text (e.g., event descriptions when unlearning a person, or vice versa).
• Real — range: [0, 1]
  • POPE metric evaluating the preservation of factual image content and visual grounding after unlearning.
• World Fact — range: [0, 1]
  • POPE metric evaluating the preservation of general world knowledge and reasoning capabilities post-unlearning.

Input / output format

Input: Multimodal instances containing an image and a text prompt/question targeting either a specific person or event depicted in the image.

Output: Text response generated by the MLLM, evaluated against ground-truth answers or using automated metrics (Precision, ROUGE-L, G-Eval, POPE).

Scoring recipe

def compute_metric(predictions, gold, metric_name):
    if metric_name in ['Efficacy', 'Generality', 'Retain', 'Scope', 'Real']:
        return precision_score(predictions, gold)
    elif metric_name in ['Event Efficacy', 'Event Generality']:
        return g_eval_score(predictions, gold)
    elif metric_name in ['Real', 'World Fact']:
        return poep_score(predictions, gold)
    return 0.0

Common pitfalls

• Cross-concept interference: unlearning a person often degrades event recognition and vice versa due to semantic coupling in the same image.
• Trade-off sensitivity: increasing unlearning steps or forget set size (5% to 15%) consistently improves efficacy but sharply degrades utility metrics like Retain and Scope.
• Early stopping is guided by training loss rather than validation metrics, which may lead to suboptimal utility preservation.

Evidence (verbatim from paper)

The performance metrics include Efficacy, Generality, Retain, Real, and World Fact. A higher score represents better performance.

Citation

@misc{xu2025pebench,
  title={PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models},
  author={Zhaopan Xu et al.},
  year={2025},
  note={arXiv:2503.12545}
}

• arXiv: 2503.12545