# Wmdp Eval

> Evaluates large language models' knowledge of hazardous topics in biosecurity, cybersecurity, and chemical security, as well as their general knowledge and fluency. It serves as a proxy for measuring dual-use risk and benchmarking unlearning methods. Use when the user wants to benchmark on WMDP, or asks about evaluating this task. Reports WMDP.

- Skill: `qhjqhj00/wmdp-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/wmdp-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/wmdp-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/wmdp-eval

---


# wmdp-eval

> The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning — Li et al. (2024) (arXiv:2403.03218, 2024)

## What this evaluates

Evaluates large language models' knowledge of hazardous topics in biosecurity, cybersecurity, and chemical security, as well as their general knowledge and fluency. It serves as a proxy for measuring dual-use risk and benchmarking unlearning methods.

## Datasets

- **WMDP** — total 3668; splits: test (3668)

## Metrics

- `WMDP` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions using top-1 logit selection. Lower scores indicate less hazardous knowledge.
- `MMLU` — range: percent
  - Percentage of correctly answered multiple-choice questions across college and high school subjects.
- `MT-Bench` — range: other
  - Single-evaluation mode scored by GPT-4-0613 as an automated judge.

## Input / output format

**Input**: Zero-shot multiple-choice question prompt with four options (A, B, C, D) and a trailing 'Answer:' token.

**Output**: Model generates the top logit choice (A, B, C, or D).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- WMDP scores are lower-is-better (indicated by ↓ in tables), unlike standard benchmarks where higher is better.
- MT-Bench evaluation relies on GPT-4 as an automated judge rather than exact-match or human grading.
- The benchmark uses a strict zero-shot format with a fixed prompt template; deviating from the template may alter model behavior.

## Evidence (verbatim from paper)

> We use the default lm-evaluation-harness zero-shot format for MMLU and WMDP, taking the top logit between A, B, C, and D as the answer choice. ... Table 2: Comparing base models and unlearning methods on question-answer evaluation (WMDP, MMLU) and fluency (MT-Bench). All WMDP and MMLU scores are percentage points.

## Citation

```bibtex
@misc{li2024wmdp,
  title={The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2403.03218}
}
```

- arXiv: 2403.03218