Sentinel Hallucination Eval

sentinel-hallucination-eval

Mitigating Object Hallucinations via Sentence-Level Early Intervention — Peng et al. (2025) (arXiv:2507.12455, 2025)

What this evaluates

Evaluates a sentence-level early intervention framework for reducing object hallucinations in multimodal large language models (MLLMs) while preserving or enhancing general vision-language capabilities across multiple standard benchmarks.

Datasets

• Object HalBench — total ?; splits: (unstated)
• AMBER — total ?; splits: (unstated)
• HallusionBench — total ?; splits: (unstated)
• VQAv2 — total ?; splits: (unstated)
• TextVQA — total ?; splits: (unstated)
• ScienceQA — total ?; splits: (unstated)
• MM-Vet — total ?; splits: (unstated)

Metrics

• response-level hallucination rate (Resp.) (primary) — range: percent
  • Percentage of generated responses that contain at least one hallucinated object. Calculated as (number of responses with hallucinations / total responses) * 100.
• mention-level hallucination rate (Ment.) — range: percent
  • Percentage of hallucinated object mentions relative to the total number of object mentions in the responses.
• AMBER F1 — range: other
  • F1 score computed on the discriminative part of the AMBER benchmark across six hallucination types.
• CHAIR — range: percent
  • Sentence-level hallucination rate measuring the proportion of sentences containing hallucinated objects.
• MM-Vet Overall — range: other
  • Composite score evaluating general vision-language capabilities, including reasoning, translation, OCR, and more.

Input / output format

Input: Image paired with a text prompt or question.

Output: Natural language response describing the image or answering the question.

Scoring recipe

def compute_metrics(predictions, golds, benchmark):
    if benchmark == 'Object_HalBench':
        resp_hall = sum(1 for p in predictions if contains_hallucination(p)) / len(predictions)
        ment_hall = count_hallucinated_mentions(predictions) / count_total_mentions(predictions)
        return resp_hall * 100, ment_hall * 100
    elif benchmark == 'AMBER':
        return f1_score(predictions, golds), accuracy(predictions, golds)
    elif benchmark == 'MM-Vet':
        return mmvet_scoring(predictions, golds)
    return standard_vqa_accuracy(predictions, golds)

Common pitfalls

• Confusing response-level vs. mention-level hallucination rates, which measure different granularities of error (per-response vs. per-mention).
• Assuming hallucination mitigation techniques inherently degrade general VQA performance; this method demonstrates concurrent improvement on benchmarks like ScienceQA and MM-Vet.
• Using out-of-domain or rewritten preference data instead of in-domain context-coherent samples, which significantly reduces effectiveness and increases hallucination rates.

Evidence (verbatim from paper)

Specifically, for the 7B model, our method achieves a 4.3 response-level (Resp.) and a 2.6 mention-level (Ment.) hallucination rate. Compared to the previous SOTA method, TPO, which achieves a 5.6 response-level and 3.2 mention-level hallucination rate, our proposed SENTINEL surpasses it by further reducing hallucinations by a total of 24% on Object Halbench.

Citation

@misc{peng2025sentinel,
  title={Mitigating Object Hallucinations via Sentence-Level Early Intervention},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2507.12455}
}

• arXiv: 2507.12455