Vib Probe Hallucination Eval

vib-probe-hallucination-eval

VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck — Zhang et al. (2026) (arXiv:2601.05547, 2026)

What this evaluates

Evaluates the ability of Vision-Language Models to avoid generating unfaithful or non-existent visual details in both closed-set discriminative QA and open-ended generative captioning. It also measures how well an auxiliary probing framework can detect these hallucinations via attention dynamics and mitigate them at inference time without retraining.

Datasets

• POPE — total 9000; splits: test (9000)
• AMBER — total 5000; splits: test (5000)
• M-HalDetect — total 16000; splits: train (12800), val (3200)
• COCO-Caption — total 2000; splits: train (1600), val (400)

Metrics

• AUPRC (primary) — range: [0, 1]
  • Area Under the Precision-Recall Curve computed across classification thresholds for hallucination probability scores.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve computed across classification thresholds.
• CHAIR (Ci, Cs) — range: percent
  • Consistency (Ci) measures the fraction of hallucinated objects in the generated text relative to ground truth. Completeness (Cs) measures the fraction of ground-truth objects missing from the generated text.
• Accuracy & F1 — range: percent
  • Standard classification metrics computed on POPE discriminative QA labels to evaluate mitigation performance.

Input / output format

Input: Image paired with a text prompt (question for discriminative tasks, or instruction for generative captioning).

Output: Model generates a text response. VIB-Probe outputs a hallucination probability score derived from compressed attention latent states.

Scoring recipe

def compute_detection_metrics(scores, labels):
    prec, rec, _ = precision_recall_curve(labels, scores)
    auprc = auc(rec, prec)
    fpr, tpr, _ = roc_curve(labels, scores)
    auroc = auc(fpr, tpr)
    return auprc, auroc

def compute_mitigation_metrics(resp, gold):
    gen_objs = extract_entities(resp)
    gt_objs = gold.objects
    ci = len(gen_objs - gt_objs) / max(len(gen_objs), 1)
    cs = len(gt_objs - gen_objs) / max(len(gt_objs), 1)
    acc = accuracy_score(gold.labels, model.predict(image, prompt))
    f1 = f1_score(gold.labels, model.predict(image, prompt))
    return ci, cs, acc, f1

Common pitfalls

• Using the full 14,216-query AMBER dataset instead of the 5,000-query subset actually used in experiments.
• Confusing detection metrics (AUPRC/AUROC) with generation quality metrics (CHAIR/F1), as they evaluate different pipeline stages.
• Calculating CHAIR metrics without normalizing by the number of generated or ground-truth objects, which fundamentally changes Ci and Cs values.
• Applying image perturbations (rotation, blur, brightness) during training rather than evaluation-only, which violates the robustness protocol.

Evidence (verbatim from paper)

To assess detection performance, we report AUPRC and AUROC Davis and Goadrich ([2006]). For generative evaluation, we utilized the CHAIR Rohrbach et al. ([2018]) metric, which quantifies object-level hallucinations by cross-referencing generated entities against ground-truth object lists. For POPE, we reported the Accuracy and F1 score metrics.

Citation

@misc{zhang2026vibprobe,
  title={VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2601.05547}
}

• arXiv: 2601.05547