Multilingual Hallucination Eval

multilingual-hallucination-eval

Mitigating Multilingual Hallucination in Large Vision-Language Models — Xiaoye Qu et al. (2024) (arXiv:2408.00550, 2024)

What this evaluates

Evaluates large vision-language models' ability to accurately describe images and answer questions without hallucinating objects or attributes across 13 languages. It probes cross-lingual alignment, instruction following, and hallucination mitigation in both discriminative and generative settings.

Datasets

• POPE MUL — total 27000; splits: test (27000)
• MME MUL — total ?; splits: test (-1)
• AMBER MUL — total ?; splits: test (-1)

Metrics

• Accuracy, Precision, Recall, F1 (primary) — range: [0, 1]
  • Standard classification metrics for object hallucination detection. Accuracy explicitly flags irrelevant or non-answers as incorrect. F1 is computed from Precision and Recall.
• ACC, ACC+, Total Score (primary) — range: other
  • Task-level accuracy (ACC) and a stricter variant (ACC+). Total Score is the sum of ACC and ACC+ across all MME perception and cognition tasks.
• CHAIR, Cover, Hal, Qualified Content (primary) — range: [0, 1]
  • CHAIR: frequency of hallucinatory objects in responses. Cover: object coverage ratio. Hal: proportion of responses containing hallucinations. Qualified Content (QC): ratio of generated sentences matching the target language, detected via langdetect.

Input / output format

Input: Image paired with a multilingual query/question (English + 12 translated languages: ru, de, zh, ja, fr, es, pt, uk, bg, tr, ar, ko).

Output: Text response generated via greedy decoding (temperature=0). For POPE/MME: short answer or selection. For AMBER: descriptive sentence(s).

Scoring recipe

def score_pope(pred, gold):
    is_correct = (pred == gold) or (is_relevant(pred) and matches(pred, gold))
    # Irrelevant answers flagged as incorrect
    acc = sum(is_correct) / len(gold)
    prec, rec, f1 = compute_precision_recall_f1(is_correct)

def score_mme(preds, golds):
    task_acc = [compute_accuracy(p, g) for p, g in zip(preds, golds)]
    total_score = sum(task_acc + task_acc_strict)

def score_amber(response, target_lang):
    qc = sum(langdetect(s).matches(target_lang) for s in response) / len(response)
    chair = count_hallucinated_objects(response) / total_objects
    hal = sum(1 for r in responses if contains_hallucination(r)) / len(responses)
    cover = compute_object_coverage(response)

Common pitfalls

• Irrelevant or non-answers in POPE are explicitly flagged as incorrect, which disproportionately affects non-English evaluations where models often output language-agnostic tokens.
• Translation artifacts in multilingual benchmarks can cause false negatives if the model answers correctly but in a slightly different phrasing than the translated gold.
• Greedy decoding (temperature=0) is strictly enforced; using sampling or higher temperatures breaks fair comparison across models.

Evidence (verbatim from paper)

Following previous works [[40], [41]] mitigating hallucinations in LVLMs, in our experiments, we adopt two widely-used discriminative benchmarks POPE and MME. In addition, to analyze the performance of our method on generative tasks, we further employ a generative benchmark AMBER for evaluation. ... The evaluation is based on four vital metrics: Accuracy, Precision, Recall, and the F1 score. Significantly, in the computation of accuracy, the model dispensing irrelevant answers is flagged as incorrect. ... Here CHAIR metric [[48]] measures the frequency of hallucinatory objects appearing in the responses. Cover measures the object coverage of responses, and Hal represents the proportion of responses with hallucinations. ... we further devise a new metric “Qualified Content” which indicates the ratio of generated sentences that align with the target language.

Citation

@misc{qu2024mitigatingmultilingualhallucination,
  title={Mitigating Multilingual Hallucination in Large Vision-Language Models},
  author={Xiaoye Qu et al. (2024)},
  year={2024},
  note={arXiv:2408.00550}
}

• arXiv: 2408.00550