# Detailverifybench Eval

> Evaluates multimodal large language models' ability to pinpoint erroneous content at the token level within long-form image captions. It probes whether models can distinguish between visually grounded facts and hallucinated details by localizing specific tokens that contradict the input image. Use when the user wants to benchmark on DetailVerifyBench, or asks about evaluating this task. Reports token-level F1.

- Skill: `qhjqhj00/detailverifybench-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/detailverifybench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/detailverifybench-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/detailverifybench-eval

---


# detailverifybench-eval

> DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions — Wang et al. (2026) (arXiv:2604.05623, 2026)

## What this evaluates

Evaluates multimodal large language models' ability to pinpoint erroneous content at the token level within long-form image captions. It probes whether models can distinguish between visually grounded facts and hallucinated details by localizing specific tokens that contradict the input image.

## Datasets

- **DetailVerifyBench** — total 1000; splits: test (1000)

## Metrics

- `token-level F1` **(primary)** — range: [0, 1]
  - Computed over token indices: Precision = |H_hat ∩ H| / |H_hat|, Recall = |H_hat ∩ H| / |H|, F1 = 2PR/(P+R). H is the set of ground-truth hallucinated token indices, H_hat is the predicted set.

## Input / output format

**Input**: An input image x and a candidate caption c consisting of N tokens.

**Output**: An augmented sequence o that preserves the original caption's word order and content, with all hallucinated tokens wrapped in <HALLUCINATION> and </HALLUCINATION> tags.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    intersection = len(set(predictions) & set(gold))
    precision = intersection / len(predictions) if predictions else 0
    recall = intersection / len(gold) if gold else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1
```

## Common pitfalls

- Models may exploit linguistic plausibility or internal text inconsistencies rather than verifying against the image, leading to inflated scores on synthetic hallucinations.
- The strict lexical faithfulness constraint requires the output to exactly match the original caption's tokens and order; any paraphrasing or reordering violates the evaluation protocol.
- Token-level evaluation differs significantly from sentence-level evaluation; a single hallucinated token makes the whole sentence hallucinated, but sentence-level metrics aggregate over binary labels.

## Evidence (verbatim from paper)

> We cast hallucination localization as a constrained text generation problem. Formally, let $x$ denote an input image and let $c\=(c_{1},c_{2},\dots,c_{N})$ a candidate caption consisting of $N$ tokens. The localization model $\pi_{\theta}$ is tasked with generating an augmented output sequence $o$ that satisfies two constraints simultaneously: ... Hallucination Localization. The model must find all hallucinated tokens and wrap them in boundary tags <HALLUCINATION> and </HALLUCINATION>. ... performance is measured by token-level Precision, Recall, and F1: $P\=\frac{|\hat{\mathcal{H}}\cap\mathcal{H}|}{|\hat{\mathcal{H}}|},\quad R\=\frac{|\hat{\mathcal{H}}\cap\mathcal{H}|}{|\mathcal{H}|},\quad F_{1}\=\frac{2PR}{P+R}.$

## Citation

```bibtex
@misc{wang2026detailverifybench,
  title={DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.05623}
}
```

- arXiv: 2604.05623