Detailverifybench Eval

detailverifybench-eval

DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions — Wang et al. (2026) (arXiv:2604.05623, 2026)

What this evaluates

Evaluates multimodal large language models' ability to pinpoint erroneous content at the token level within long-form image captions. It probes whether models can distinguish between visually grounded facts and hallucinated details by localizing specific tokens that contradict the input image.

Datasets

• DetailVerifyBench — total 1000; splits: test (1000)

Metrics

• token-level F1 (primary) — range: [0, 1]
  • Computed over token indices: Precision = |H_hat ∩ H| / |H_hat|, Recall = |H_hat ∩ H| / |H|, F1 = 2PR/(P+R). H is the set of ground-truth hallucinated token indices, H_hat is the predicted set.

Input / output format

Input: An input image x and a candidate caption c consisting of N tokens.

Output: An augmented sequence o that preserves the original caption's word order and content, with all hallucinated tokens wrapped in and tags.

Scoring recipe

def compute_metrics(predictions, gold):
    intersection = len(set(predictions) & set(gold))
    precision = intersection / len(predictions) if predictions else 0
    recall = intersection / len(gold) if gold else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• Models may exploit linguistic plausibility or internal text inconsistencies rather than verifying against the image, leading to inflated scores on synthetic hallucinations.
• The strict lexical faithfulness constraint requires the output to exactly match the original caption's tokens and order; any paraphrasing or reordering violates the evaluation protocol.
• Token-level evaluation differs significantly from sentence-level evaluation; a single hallucinated token makes the whole sentence hallucinated, but sentence-level metrics aggregate over binary labels.

Evidence (verbatim from paper)

We cast hallucination localization as a constrained text generation problem. Formally, let $x$ denote an input image and let $c=(c_{1},c_{2},\dots,c_{N})$ a candidate caption consisting of $N$ tokens. The localization model $\pi_{\theta}$ is tasked with generating an augmented output sequence $o$ that satisfies two constraints simultaneously: ... Hallucination Localization. The model must find all hallucinated tokens and wrap them in boundary tags and . ... performance is measured by token-level Precision, Recall, and F1: $P=\frac{|\hat{\mathcal{H}}\cap\mathcal{H}|}{|\hat{\mathcal{H}}|},\quad R=\frac{|\hat{\mathcal{H}}\cap\mathcal{H}|}{|\mathcal{H}|},\quad F_{1}=\frac{2PR}{P+R}.$

Citation

@misc{wang2026detailverifybench,
  title={DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.05623}
}

• arXiv: 2604.05623