Shale Eval

shale-eval

SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs — Bei Yan et al. (2025) (arXiv:2508.09584, 2025)

What this evaluates

This benchmark evaluates fine-grained hallucination in Large Vision-Language Models (LVLMs) by testing their faithfulness to visual inputs and factuality against external knowledge. It measures model performance under clean conditions and across hierarchical input perturbations (image, instruction, and combination levels) to assess hallucination resistance.

Datasets

• SHALE — total 30000; splits: test (30000); repo https://github.com/BeiiiY/SHALE

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions on discriminative tasks (yes-or-no, multiple-choice, free-form).
• non-hallucination rate (primary) — range: [0, 1]
  • Proportion of model responses deemed hallucination-free by an LLM-as-a-Judge, given the image, instruction, and ground truth.
• Resistance Rate (RR) — range: [0, 1]
  • Measures robustness to perturbations: RR(θ) = Σ[1_nh(i,t,θ) · 1_nh(ĩ,t̃,θ)] / Σ[1_nh(i,t,θ)], where 1_nh is 1 if the response is non-hallucinated for clean (i,t) or perturbed (ĩ,t̃) inputs.

Input / output format

Input: Paired image and text instruction. Inputs may be clean or perturbed at the image level (style transformation, corruption, adversarial noise, scene text injection), instruction level (confusing synonyms, misleading prefixes), or combination level.

Output: Text response from the LVLM. For discriminative tasks: yes/no, selected option, or free-form answer. For generative tasks: image caption or description.

Scoring recipe

# For discriminative tasks
acc = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)

# For generative tasks (LLM-as-Judge)
nh_flags = [1 if judge_llm(image, instruction, pred, gold) == "non-hallucinated" else 0 for pred in predictions]
non_hall_rate = sum(nh_flags) / len(nh_flags)

# Resistance Rate (RR)
clean_nh = [1 if judge_llm(img, inst, pred, gold) == "non-hallucinated" else 0 for pred in clean_preds]
perturbed_nh = [1 if judge_llm(p_img, p_inst, p_pred, gold) == "non-hallucinated" else 0 for p_pred in perturbed_preds]
rr = sum(c * p for c, p in zip(clean_nh, perturbed_nh)) / sum(clean_nh)

Common pitfalls

• Failing to distinguish between faithfulness hallucinations (visual perception errors) and factuality hallucinations (knowledge errors), which require different evaluation templates and knowledge bases.
• Reporting only clean-condition accuracy without evaluating the hierarchical perturbation scenarios (image, instruction, combination), which obscures the model's true hallucination resistance.
• Using object-level detection metrics like CHAIR instead of the proposed LLM-as-a-Judge non-hallucination rate, which limits generalizability across diverse hallucination types and task formats.

Evidence (verbatim from paper)

For discriminative tasks, we adopt accuracy as the evaluation metric. For generative tasks, we follow previous work, employing an LLM-as-a-Judge approach to assess hallucinations. Specifically, given the image content, instruction, and ground-truth answer, we prompt an advanced LLM to determine whether the model’s response is hallucination-free, and calculate the non-hallucination rate as the evaluation metric. Compared to CHAIR, which focuses solely on object-level hallucination detection, the LLM-as-a-Judge approach offers broader generalizability by supporting diverse hallucination types and task formats. Additionally, the non-hallucination rate is a comparable metric to accuracy, enabling direct derivation of an overall average performance score. To quantify model resistance to perturbations under various hallucination-inducing noisy scenarios, we propose the Resistance Rate (RR) metric. It measures the proportion of input image-text pairs with non-hallucinated responses under clean scenario that remain non-hallucinated when subject to perturbations.

Citation

@misc{yan2025shale,
  title={SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs},
  author={Bei Yan et al. (2025)},
  year={2025},
  note={arXiv:2508.09584}
}

• arXiv: 2508.09584