qhjqhj00 llava-scissor-eval
LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs — Sun et al. (2025) (arXiv:2506.21862, 2025)
What this evaluates
Evaluates the effectiveness of a training-free token compression method for video large language models across various video understanding tasks, including QA, long-video understanding, and multi-choice benchmarks, under different token retention ratios.
Datasets
- ActivityNet-QA — total ?; splits: test (-1)
- Video-ChatGPT — total ?; splits: test (-1)
- Next-QA — total ?; splits: test (-1)
- Egoschema — total ?; splits: test (-1)
- MLVU — total ?; splits: test (-1)
- Video-MME — total ?; splits: test (-1)
- VideoMMMU — total ?; splits: test (-1)
- MVBench — total ?; splits: test (-1)
Metrics
Avg.(%)(primary) — range: percent- Percentage of correct answers on each benchmark. The headline metric is the average accuracy across all evaluated benchmarks.
Input / output format
Input: Video frames processed by a vision encoder (SIGLIP) and a text question/prompt.
Output: Text answer generated by the LLM (Qwen 2.5).
Scoring recipe
def compute_avg_accuracy(predictions, gold_answers):
correct = sum(1 for pred, gold in zip(predictions, gold_answers) if pred == gold)
return (correct / len(gold_answers)) * 100
Common pitfalls
- Token retention ratio is controlled by a similarity threshold $\tau$, so the exact number of retained tokens varies across datasets with different video distributions.
- Comparisons are only fair when using the same modified base model (LLaVA-OneVision with SIGLIP and Qwen 2.5), not the original LLaVA-OneVision.
- FLOPs are calculated only for the LLM stage (prefilling and decoding), excluding vision encoder and compression overhead.
Evidence (verbatim from paper)
| Method | Retention ratio | EgoSchema | MLVU | VideoMME | VideoMMMU | Avg.(%) |
Citation
@misc{sun2025llavascissor,
title={LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs},
author={Sun et al. (2025)},
year={2025},
note={arXiv:2506.21862}
}
- arXiv: 2506.21862