# Pebench Eval

> Evaluates multimodal large language models on their ability to selectively forget specific person or event concepts while preserving general knowledge. It probes cross-concept interference, unlearning efficacy, and the trade-off between forgetting targeted data and maintaining model utility. Use when the user wants to benchmark on PEBench, or asks about evaluating this task. Reports Efficacy.

- Skill: `qhjqhj00/pebench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/pebench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/pebench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/pebench-eval

---


# pebench-eval

> PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models — Zhaopan Xu et al. (arXiv:2503.12545, 2025)

## What this evaluates

Evaluates multimodal large language models on their ability to selectively forget specific person or event concepts while preserving general knowledge. It probes cross-concept interference, unlearning efficacy, and the trade-off between forgetting targeted data and maintaining model utility.

## Datasets

- **PEBench** — total 8000; splits: forget (-1), retain (-1)

## Metrics

- `Efficacy` **(primary)** — range: [0, 1]
  - Precision measuring the proportion of correctly unlearned target samples (person or event). Higher values indicate successful forgetting.
- `Generality` — range: [0, 1]
  - Precision measuring unlearning success on related or unseen samples of the target concept, testing generalization of the forgetting process.
- `Retain` — range: [0, 1]
  - Precision measuring the preservation of non-target knowledge on the retain set after unlearning.
- `Scope` — range: [0, 1]
  - ROUGE-L score measuring the preservation of related descriptive text (e.g., event descriptions when unlearning a person, or vice versa).
- `Real` — range: [0, 1]
  - POPE metric evaluating the preservation of factual image content and visual grounding after unlearning.
- `World Fact` — range: [0, 1]
  - POPE metric evaluating the preservation of general world knowledge and reasoning capabilities post-unlearning.

## Input / output format

**Input**: Multimodal instances containing an image and a text prompt/question targeting either a specific person or event depicted in the image.

**Output**: Text response generated by the MLLM, evaluated against ground-truth answers or using automated metrics (Precision, ROUGE-L, G-Eval, POPE).

## Scoring recipe

```python
def compute_metric(predictions, gold, metric_name):
    if metric_name in ['Efficacy', 'Generality', 'Retain', 'Scope', 'Real']:
        return precision_score(predictions, gold)
    elif metric_name in ['Event Efficacy', 'Event Generality']:
        return g_eval_score(predictions, gold)
    elif metric_name in ['Real', 'World Fact']:
        return poep_score(predictions, gold)
    return 0.0
```

## Common pitfalls

- Cross-concept interference: unlearning a person often degrades event recognition and vice versa due to semantic coupling in the same image.
- Trade-off sensitivity: increasing unlearning steps or forget set size (5% to 15%) consistently improves efficacy but sharply degrades utility metrics like Retain and Scope.
- Early stopping is guided by training loss rather than validation metrics, which may lead to suboptimal utility preservation.

## Evidence (verbatim from paper)

> The performance metrics include Efficacy, Generality, Retain, Real, and World Fact. A higher score represents better performance.

## Citation

```bibtex
@misc{xu2025pebench,
  title={PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models},
  author={Zhaopan Xu et al.},
  year={2025},
  note={arXiv:2503.12545}
}
```

- arXiv: 2503.12545