Vtc R1 Eval

vtc-r1-eval

VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning — Yibo Wang et al. (2026) (arXiv:2601.22069, 2026)

What this evaluates

This evaluation protocol assesses a vision-language model's ability to perform long-context mathematical and scientific reasoning using vision-text compression. It measures both reasoning accuracy across diverse benchmarks and computational efficiency in terms of token usage and inference latency.

Datasets

• GSM8K — total ?; splits: test (-1)
• MATH500 — total ?; splits: test (-1)
• AIME25 — total ?; splits: test (-1)
• AMC23 — total ?; splits: test (-1)
• GPQA-Diamond — total ?; splits: test (-1)

Metrics

• Accuracy (ACC) (primary) — range: percent
  • For GSM8K, MATH500, and GPQA-Diamond, pass@1 accuracy is reported. For AIME25 and AMC23, 16 responses are generated per problem and avg@16 accuracy is reported.
• Token (TOK) — range: other
  • The average number of tokens in the generated responses.
• Latency (LAT) — range: other
  • Average inference latency per generation, computed as (t2 - t1) / (m * n), where t1 and t2 are wall-clock timestamps for the entire inference process, m is the number of problems, and n is the number of generations per problem.

Input / output format

Input: Mathematical or scientific reasoning problems (questions).

Output: Generated reasoning traces followed by a final answer.

Scoring recipe

def compute_acc(predictions, golds, n=1):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    return correct / len(golds) if n == 1 else sum(correct for _ in range(n)) / (len(golds) * n)

def compute_tok(predictions):
    return sum(len(tokenizer.encode(p)) for p in predictions) / len(predictions)

# LAT = (t2 - t1) / (m * n) measured via wall-clock timestamps

Common pitfalls

• Different benchmarks use different sampling strategies (pass@1 vs avg@16), which can mislead direct accuracy comparisons if not accounted for.
• Latency (LAT) is measured as wall-clock time per generation on specific hardware/frameworks (vLLM), making it non-portable without exact hardware specs.
• Token count (TOK) only measures generated tokens, ignoring the compressed visual tokens used as context, which may understate total computational cost.

Evidence (verbatim from paper)

We employ the following three metrics to evaluate the model’s performance. • Accuracy (ACC): For GSM8K, MATH500, and GPQA-Diamond, we report pass@1 accuracy. For AIME25 and AMC23, due to their limited dataset sizes, we generate 16 responses per problem and report avg@16 accuracy. • Token (TOK): The average number of tokens in the generated responses. • Latency (LAT): We measure the average inference latency per generation. Given a dataset with $m$ problems, where each problem is generated $n$ times (e.g., $n=16$ for AIME25 and AMC23), let $t_{1}$ and $t_{2}$ denote the wall-clock timestamps at the start and end of the entire inference process, respectively. The latency is computed as: $LAT=\frac{t_{2}-t_{1}}{m\times n}.$

Citation

@misc{wang2026vtcr1,
  title={VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning},
  author={Yibo Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.22069}
}

• arXiv: 2601.22069