qhjqhj00 mm-inference-eval
MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention — Li et al. (2025) (arXiv:2504.16083, 2025)
What this evaluates
This evaluation probes the effectiveness and efficiency of dynamic sparse attention methods for long-context vision-language models (VLMs). It tests the model's ability to perform long-video understanding, retrieve specific visual or mixed-modality information from extremely long contexts (Needle in a Haystack), and maintain accuracy while reducing computational cost and latency.
Datasets
- Video Understanding Benchmarks — total ?; splits: test (-1), val (-1)
- V-NIAH — total ?; splits: test (-1)
- MM-NIAH — total ?; splits: test (-1)
Metrics
task accuracy / official benchmark score(primary) — range: percent- Uses the official metrics and scripts provided by each benchmark (ActNet-QA, EgoSchema, Next-QA, PerceptionTest, VideoDC, VideoMME). Typically exact-match or multiple-choice accuracy for QA tasks, and standard captioning/retrieval scores where applicable.
end-to-end latency— range: other- Wall-clock time measured for the full pre-filling/inference pass on a single NVIDIA A100 using bfloat16 and greedy decoding. Reported in seconds or milliseconds.
kernel-level latency— range: other- Wall-clock time measured specifically for the attention kernel computation, used to isolate the speedup from sparse attention patterns.
Input / output format
Input: Long-video inputs ranging from 110 to 4.5k frames (up to ~1.1M tokens), paired with text queries for QA, captioning, or retrieval tasks. MM-NIAH inputs additionally contain 25% text segments inserted at the document level across different frames.
Output: Model-generated answers or selected choices for QA/retrieval tasks. Latency is measured as wall-clock time per pass.
Scoring recipe
```python
# For accuracy metrics
import official_benchmark_scripts
predictions = model.generate(input_video, input_query)
gold_answers = load_gold_answers(dataset)
metric_value = official_benchmark_scripts.compute_score(predictions, gold_answers)
# For latency
import time
torch.cuda.synchronize()
start = time.time()
model.generate(input_video, input_query) # or model.prefill()
torch.cuda.synchronize()
latency = time.time() - start
## Common pitfalls
- Static sparse patterns (e.g., A-shape, Tri-shape) show notable performance drops on multi-choice VQA tasks like EgoSchema, even when FLOPs are matched.
- Sparse indices trained on textual contexts fail to generalize to visual modalities; modality-aware permutation is required to maintain performance across mixed-modality boundaries.
- Latency measurements must use greedy decoding and bfloat16 on an A100 to ensure stable, comparable results as specified in the setup.
## Evidence (verbatim from paper)
> We evaluate our method on three general long-video tasks: long-video understanding, Video Needle in a Haystack, and Video-Text Needle in a Haystack. Our evaluation uses the official metrics and scripts provided by these tasks. Latency experiments are performed on a single NVIDIA A100 using bfloat16, with greedy decoding to ensure stable results.
## Citation
```bibtex
@misc{li2025mm inference,
title={MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention},
author={Li et al. (2025)},
year={2025},
note={arXiv:2504.16083}
}
- arXiv: 2504.16083