M2cqa Eval

m2cqa-eval

Once Correct, Still Wrong: Counterfactual Hallucination in Multilingual Vision-Language Models — Basel Mousi et al. (arXiv:2602.05437, 2026)

What this evaluates

Evaluates vision-language models' ability to correctly identify true statements about images while rejecting culturally plausible but visually incorrect counterfactual statements. It specifically probes grounding failures and cultural reasoning biases across multiple languages and dialects.

Datasets

• M²CQA — total ?; splits: test (-1)

Metrics

• CFHR (primary) — range: [0, 1]
  • Counterfactual Hallucination Rate. Measures the proportion of counterfactual statements incorrectly accepted as True, conditional on the model having correctly identified the corresponding true statement. CFHR = (Counterfactuals predicted as True) / (True statements correctly predicted as True).
• Q+ — range: [0, 1]
  • Accuracy on true statements. Proportion of true statements correctly identified as True.
• Q- — range: [0, 1]
  • Accuracy on counterfactual statements. Proportion of counterfactual statements correctly identified as False.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for the positive class (True). Summarizes overall binary classification accuracy.

Input / output format

Input: An image paired with a statement (either a true description or a culturally plausible counterfactual) in English, Modern Standard Arabic (MSA), or an Arabic dialect (Levantine/Egyptian).

Output: A binary True/False answer, optionally followed by a justification or reasoning trace depending on the prompting setting.

Scoring recipe

def compute_cfhr(predictions, gold_labels):
    # predictions: 1 if model says True, 0 if False
    # gold_labels: 1 for true statements, 0 for counterfactuals
    true_correct_count = sum(1 for p, g in zip(predictions, gold_labels) if g == 1 and p == 1)
    if true_correct_count == 0:
        return 0.0
    counterfactual_accepted = sum(1 for p, g in zip(predictions, gold_labels) if g == 0 and p == 1)
    return counterfactual_accepted / true_correct_count

Common pitfalls

• High Q+ accuracy can mask severe counterfactual hallucination (high CFHR), making standard accuracy metrics misleading.
• Reasoning-first prompting can exacerbate hallucination by shifting decision criteria from visual evidence to cultural priors.
• Low CFHR may indicate overly conservative rejection rather than strong visual grounding, so it must be interpreted alongside Q+.

Evidence (verbatim from paper)

The proposed Counterfactual Hallucination Rate (CFHR) isolates this conditional failure by measuring hallucination only in cases where the model has already succeeded on Q+.

Citation

@misc{mousi2026m2cqa,
  title={Once Correct, Still Wrong: Counterfactual Hallucination in Multilingual Vision-Language Models},
  author={Basel Mousi et al.},
  year={2026},
  note={arXiv:2602.05437}
}

• arXiv: 2602.05437