qhjqhj00 visual-counterfact-eval
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts — Nooralahzadeh et al. (2026) (arXiv:2604.09364, 2026)
What this evaluates
Evaluates how vision-language models resolve conflicts between visual input and language priors by reasoning about altered visual attributes (color and size). It probes whether models rely on visual evidence or textual priors when they contradict.
Datasets
- Visual-Counterfact — total 1220; splits: Color (493), Size (727)
Metrics
MAC(primary) — range: layer index- Mean crossover layer: the average transformer layer index where the visual logit probability first surpasses the language-prior logit probability.
R%— range: percent- Percentage of samples where the visual logit wins over the prior logit at the final output layer.
D%— range: percent- MAC depth normalized as a percentage of the model's total layers L.
Input / output format
Input: An image containing a counterfactual visual attribute (e.g., a blue banana or flipped object sizes) paired with a text prompt querying that attribute.
Output: A text response from the VLM indicating the perceived attribute (e.g., 'blue', 'large'), derived from final-layer logits.
Scoring recipe
def compute_metrics(predictions, golds, logit_sequence, total_layers):
# R%: % samples where visual logit wins at final layer
visual_wins = sum(1 for p in predictions if logit_sequence[-1][p] > logit_sequence[-1][prior])
r_pct = (visual_wins / len(predictions)) * 100
# MAC: Mean crossover layer (where visual logit surpasses prior)
crossover_layers = []
for i, logits in enumerate(logit_sequence):
if logits[visual_attr] > logits[prior_attr]:
crossover_layers.append(i)
break
mac = sum(crossover_layers) / len(crossover_layers) if crossover_layers else -1
d_pct = (mac / total_layers) * 100
return {'R%': r_pct, 'MAC': mac, 'D%': d_pct}
Common pitfalls
- Confusing early-layer visual encoding with final-output grounding: models often correctly perceive visual features in shallow layers but fail to output them due to language priors overriding at the end.
- Assuming model scaling guarantees better arbitration: larger parameter counts (e.g., 72B) do not consistently improve R% or reduce MAC compared to 7B variants.
- Comparing raw crossover layers across architectures: MAC values are highly architecture-dependent, requiring normalization by total layers (D%) for fair cross-model comparison.
Evidence (verbatim from paper)
We use the Visual-Counterfact dataset (Golovanevsky et al., 2025a), which provides images with visually altered properties to create controlled visual-linguistic conflicts. The dataset contains two distinct splits, each corresponding to a specific visual attribute reasoning task: Color (493 examples, like a banana that’s blue instead of yellow) and Size (727 examples, where the usual size relationships between objects are flipped. MAC = mean crossover layer; R% = % samples where visual logit wins at final layer; D% = MAC depth as % of total layers L.
Citation
@misc{nooralahzadeh2026arbitration,
title={Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts},
author={Nooralahzadeh et al. (2026)},
year={2026},
note={arXiv:2604.09364}
}
- arXiv: 2604.09364