qhjqhj00 vtcbench-eval
QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression — Li et al. (2026) (arXiv:2603.21232, 2026)
What this evaluates
Evaluates the ability of visual token compression methods to retain task-relevant visual information across five dimensions: global understanding, spatial and counting, reasoning and common sense, style and emotion, and local details.
Datasets
- VTCBench — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) × 100.
Input / output format
Input: An image paired with a textual question or instruction.
Output: A generated text response.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
return (correct / len(gold)) * 100
Common pitfalls
- Performance is highly sensitive to the number of retained visual tokens; comparisons must fix token counts (e.g., 144 vs 192) to ensure fairness.
- Different compression strategies inherently favor different task types (e.g., pruning for local details, pooling for global context), so reporting only a single average can mask trade-offs.
Evidence (verbatim from paper)
We evaluate our model on ten representative public vision-understanding benchmarks, as well as on our newly constructed VTCBench. ... Table 3 reports the accuracy of different methods, where a higher score indicates better retention of visual information.
Citation
@misc{li2026qmoP,
title={QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression},
author={Li et al. (2026)},
year={2026},
note={arXiv:2603.21232}
}
- arXiv: 2603.21232