# Vtcbench Eval

> vtcbench-eval

- Skill: `qhjqhj00/vtcbench-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/vtcbench-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vtcbench-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/vtcbench-eval

---


# vtcbench-eval

> QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression — Li et al. (2026) (arXiv:2603.21232, 2026)

## What this evaluates

Evaluates the ability of visual token compression methods to retain task-relevant visual information across five dimensions: global understanding, spatial and counting, reasoning and common sense, style and emotion, and local details.

## Datasets

- **VTCBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) × 100.

## Input / output format

**Input**: An image paired with a textual question or instruction.

**Output**: A generated text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Performance is highly sensitive to the number of retained visual tokens; comparisons must fix token counts (e.g., 144 vs 192) to ensure fairness.
- Different compression strategies inherently favor different task types (e.g., pruning for local details, pooling for global context), so reporting only a single average can mask trade-offs.

## Evidence (verbatim from paper)

> We evaluate our model on ten representative public vision-understanding benchmarks, as well as on our newly constructed VTCBench. ... Table 3 reports the accuracy of different methods, where a higher score indicates better retention of visual information.

## Citation

```bibtex
@misc{li2026qmoP,
  title={QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.21232}
}
```

- arXiv: 2603.21232