Ghost 100 Eval

ghost-100-eval

LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models — Jiang et al. (2026) (arXiv:2604.18803, 2026)

What this evaluates

This benchmark probes the susceptibility of vision-language models to tone-induced hallucination under controlled negative-ground-truth conditions. It isolates linguistic prompt intensity as the sole variable to measure both the frequency and severity of unsupported content generation when visual evidence is deliberately absent or illegible.

Datasets

• Ghost-100 — total 800; splits: test (800)

Metrics

• H-Rate (primary) — range: percent
  • Proportion of model responses that commit to unsupported target-specific content under a negative-ground-truth setting, computed via a rule-based whitelist policy.
• H-Score — range: other
  • Discrete severity score (1–5) assigned by a GPT-4o-mini judge evaluating the degree of unsupported linguistic commitment in the model response. Levels ≥ 3 are treated as hallucinated for binary summaries.

Input / output format

Input: A single image paired with a text prompt. Prompts are systematically varied across five escalating tone levels (from permissive queries to strong directive pressure) while the image and task identity remain fixed.

Output: A natural language text response from the VLM addressing the prompt.

Scoring recipe

# H-Rate (Rule-based)
h_rate = sum(1 for r in responses if rule_based_policy(r, target_absent=True)) / len(responses) * 100

# H-Score (LLM-as-Judge)
h_scores = [gpt4o_mini_judge(p, r) for p, r in zip(prompts, responses)]
h_score_avg = sum(h_scores) / len(h_scores)

Common pitfalls

• Treating hallucination as a single binary failure mode rather than analyzing both frequency (H-Rate) and severity (H-Score).
• Assuming hallucination intensity increases monotonically with prompt tone; results show non-monotonic, task-dependent sensitivity.
• Confusing reasonable inference with hallucination in ambiguous natural images (mitigated here by design constraints).

Evidence (verbatim from paper)

To quantify how often hallucination occurs under different prompt tones, we measure hallucination rate (H-Rate) using a rule-based policy. The H-Rate policy was initially calibrated through manual inspection of model responses, which revealed that some outputs correctly reproduced visible non-target text rather than hallucinated target content. To make evaluation scalable and consistent, these manual criteria were then formalized into a whitelist-based, category-aware H-Rate policy. In our negative-ground-truth setting, the queried target is absent, unreadable, blank, or otherwise not visually determinable by construction; therefore, a response is labeled as hallucinated when it commits to unsupported target-specific content. Hallucination rate is computed as defined in Eq.[1]. All nine models contribute the full 4,000 valid H-Rate responses in the main benchmark summary.

Citation

@misc{jiang2026ghost100,
  title={LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.18803}
}

• arXiv: 2604.18803