qhjqhj00 pixel-reasoner-eval
Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning — Wang et al. (2025) (arXiv:2505.15966, 2025)
What this evaluates
Evaluates multimodal models' ability to perform fine-grained visual reasoning, object counting, temporal video understanding, and complex infographic parsing. It specifically probes whether models can effectively leverage pixel-space operations (e.g., zooming, frame selection) rather than defaulting to text-only reasoning pathways.
Datasets
- V (V-Star)* — total ?; splits: V* test (-1)
- TallyQA — total ?; splits: TallyQA-complex (-1)
- MVBench — total ?; splits: MVBench-test (-1)
- InfographicVQA — total ?; splits: InfoVQA-test (-1)
Metrics
Acc(primary) — range: percent- Accuracy: the percentage of questions where the model's generated answer exactly matches the ground truth label.
ANLS— range: [0, 1]- Average Normalized Levenshtein Similarity: the mean of (1 - normalized Levenshtein distance) across all predictions and their corresponding gold answers.
Input / output format
Input: Multimodal inputs (images, webpages, or videos) paired with natural language questions/prompts.
Output: Textual answers generated via greedy decoding.
Scoring recipe
def compute_metrics(predictions, golds, metric_name):
if metric_name == 'Acc':
correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
return (correct / len(golds)) * 100
elif metric_name == 'ANLS':
total_sim = 0.0
for p, g in zip(predictions, golds):
dist = levenshtein_distance(p, g)
max_len = max(len(p), len(g))
total_sim += 1.0 - (dist / max_len) if max_len > 0 else 1.0
return total_sim / len(golds)
return 0.0
Common pitfalls
- Using beam search or sampling instead of greedy decoding, which the paper explicitly specifies for all evaluations.
- Evaluating on the full TallyQA dataset instead of the 'complex' split, which significantly changes difficulty and reported scores.
- Ignoring that several baseline models only report results on V* Bench, making cross-benchmark comparisons for those models incomplete.
Evidence (verbatim from paper)
| Metric | | Acc | Acc | Acc | ANLS |
Citation
@misc{wang2025pixelreasoner,
title={Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning},
author={Wang et al. (2025)},
year={2025},
note={arXiv:2505.15966}
}
- arXiv: 2505.15966