qhjqhj00 pevlm-longvideo-eval
PEVLM: Parallel Encoding for Vision-Language Models — Kang et al. (2025) (arXiv:2506.19651, 2025)
What this evaluates
Evaluates the accuracy and latency-constrained performance of vision-language models on long-video understanding tasks. It measures how well models preserve temporal reasoning and answer questions about extended video sequences under strict computational and time budgets.
Datasets
- LongVideoBench — total ?; splits: test (-1)
- VideoMME — total ?; splits: test (-1)
- EgoSchema — total ?; splits: test (-1)
- MVBench — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions out of the total dataset. In latency-constrained settings, samples exceeding the time limit are counted as failures.
latency-constrained accuracy— range: percent- Accuracy computed on samples that complete inference within a specified time threshold (e.g., 20s, 30s, 40s). Samples exceeding the limit are treated as incorrect.
speedup— range: other- Ratio of baseline inference time to PEVLM inference time, measured separately for the attention module and the full LLM.
Input / output format
Input: Video frames encoded into tokens, concatenated with a system prompt and a question. Context blocks are fixed at ~4096 tokens, with a shared sink block derived from initial frames or the system prompt.
Output: Textual answers to video understanding questions.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
return (correct / len(gold)) * 100
def compute_latency_constrained_accuracy(predictions, gold, latencies, threshold):
correct = 0
total = len(gold)
for p, g, t in zip(predictions, gold, latencies):
if t <= threshold and p.strip() == g.strip():
correct += 1
return (correct / total) * 100
Common pitfalls
- Temperature for the APE baseline must be fixed at T=1.0; lowering it further skews attention distributions and degrades accuracy.
- Context block size is strictly fixed at 4096 tokens across all methods; smaller blocks reduce accuracy while larger blocks hurt latency.
- Latency-constrained accuracy counts samples exceeding the time limit as failures (incorrect), rather than excluding them from the evaluation denominator.
Evidence (verbatim from paper)
This experiment aims to evaluate the impact of PEVLM on accuracy in long-video understanding tasks. ... we introduce a latency threshold during inference: any sample that exceeds this latency limit is treated as a failure.
Citation
@misc{kang2025pevlm,
title={PEVLM: Parallel Encoding for Vision-Language Models},
author={Kang et al. (2025)},
year={2025},
note={arXiv:2506.19651}
}
- arXiv: 2506.19651