# Multilingual Hallucination Eval

> Evaluates large vision-language models' ability to accurately describe images and answer questions without hallucinating objects or attributes across 13 languages. It probes cross-lingual alignment, instruction following, and hallucination mitigation in both discriminative and generative settings. Use when the user wants to benchmark on POPE MUL, MME MUL, AMBER MUL, or asks about evaluating this task. Reports Accuracy, Precision, Recall, F1, ACC, ACC+, Total Score, CHAIR, Cover, Hal, Qualified Content.

- Skill: `qhjqhj00/multilingual-hallucination-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/multilingual-hallucination-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/multilingual-hallucination-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/multilingual-hallucination-eval

---


# multilingual-hallucination-eval

> Mitigating Multilingual Hallucination in Large Vision-Language Models — Xiaoye Qu et al. (2024) (arXiv:2408.00550, 2024)

## What this evaluates

Evaluates large vision-language models' ability to accurately describe images and answer questions without hallucinating objects or attributes across 13 languages. It probes cross-lingual alignment, instruction following, and hallucination mitigation in both discriminative and generative settings.

## Datasets

- **POPE MUL** — total 27000; splits: test (27000)
- **MME MUL** — total ?; splits: test (-1)
- **AMBER MUL** — total ?; splits: test (-1)

## Metrics

- `Accuracy, Precision, Recall, F1` **(primary)** — range: [0, 1]
  - Standard classification metrics for object hallucination detection. Accuracy explicitly flags irrelevant or non-answers as incorrect. F1 is computed from Precision and Recall.
- `ACC, ACC+, Total Score` **(primary)** — range: other
  - Task-level accuracy (ACC) and a stricter variant (ACC+). Total Score is the sum of ACC and ACC+ across all MME perception and cognition tasks.
- `CHAIR, Cover, Hal, Qualified Content` **(primary)** — range: [0, 1]
  - CHAIR: frequency of hallucinatory objects in responses. Cover: object coverage ratio. Hal: proportion of responses containing hallucinations. Qualified Content (QC): ratio of generated sentences matching the target language, detected via langdetect.

## Input / output format

**Input**: Image paired with a multilingual query/question (English + 12 translated languages: ru, de, zh, ja, fr, es, pt, uk, bg, tr, ar, ko).

**Output**: Text response generated via greedy decoding (temperature=0). For POPE/MME: short answer or selection. For AMBER: descriptive sentence(s).

## Scoring recipe

```python
def score_pope(pred, gold):
    is_correct = (pred == gold) or (is_relevant(pred) and matches(pred, gold))
    # Irrelevant answers flagged as incorrect
    acc = sum(is_correct) / len(gold)
    prec, rec, f1 = compute_precision_recall_f1(is_correct)

def score_mme(preds, golds):
    task_acc = [compute_accuracy(p, g) for p, g in zip(preds, golds)]
    total_score = sum(task_acc + task_acc_strict)

def score_amber(response, target_lang):
    qc = sum(langdetect(s).matches(target_lang) for s in response) / len(response)
    chair = count_hallucinated_objects(response) / total_objects
    hal = sum(1 for r in responses if contains_hallucination(r)) / len(responses)
    cover = compute_object_coverage(response)
```

## Common pitfalls

- Irrelevant or non-answers in POPE are explicitly flagged as incorrect, which disproportionately affects non-English evaluations where models often output language-agnostic tokens.
- Translation artifacts in multilingual benchmarks can cause false negatives if the model answers correctly but in a slightly different phrasing than the translated gold.
- Greedy decoding (temperature=0) is strictly enforced; using sampling or higher temperatures breaks fair comparison across models.

## Evidence (verbatim from paper)

> Following previous works *[[40], [41]]* mitigating hallucinations in LVLMs, in our experiments, we adopt two widely-used discriminative benchmarks POPE and MME. In addition, to analyze the performance of our method on generative tasks, we further employ a generative benchmark AMBER for evaluation. ... The evaluation is based on four vital metrics: Accuracy, Precision, Recall, and the F1 score. Significantly, in the computation of accuracy, the model dispensing irrelevant answers is flagged as incorrect. ... Here CHAIR metric *[[48]]* measures the frequency of hallucinatory objects appearing in the responses. Cover measures the object coverage of responses, and Hal represents the proportion of responses with hallucinations. ... we further devise a new metric “Qualified Content” which indicates the ratio of generated sentences that align with the target language.

## Citation

```bibtex
@misc{qu2024mitigatingmultilingualhallucination,
  title={Mitigating Multilingual Hallucination in Large Vision-Language Models},
  author={Xiaoye Qu et al. (2024)},
  year={2024},
  note={arXiv:2408.00550}
}
```

- arXiv: 2408.00550