Flashvlm Eval

flashvlm-eval

FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models — Cai et al. (2025) (arXiv:2512.20561, 2025)

What this evaluates

Evaluates the robustness and efficiency of text-guided visual token pruning in large multimodal models. It probes whether aggressive token compression (retaining 32–128 tokens for images, 114–455 for video) degrades performance on image and video question-answering tasks, and measures cross-modal grounding quality via spatial alignment and semantic overlap metrics.

Datasets

• VQAv2 — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• ScienceQA-IMG — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MMBench-CN — total ?; splits: test (-1)
• MM Vet — total ?; splits: test (-1)
• TGIF-QA — total ?; splits: test (-1)
• MSVDQA — total ?; splits: test (-1)
• MSRVTT-QA — total ?; splits: test (-1)
• ActivityNet-QA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions across benchmarks, computed via official evaluation scripts.
• average_accuracy (primary) — range: percent
  • Mean accuracy across all 10 image benchmarks or 3 video benchmarks, used as the headline performance indicator.
• attention_distance — range: other
  • Quantifies the spatial deviation of the model's final focus region from the ground truth, capturing RoPE-induced proximity bias.
• score_map_entropy — range: other
  • Measures the concentration of the token selection score distribution; lower values indicate less redundancy and more stable salient regions.
• token_box_iou — range: other
  • Intersection over Union between selected visual tokens and manually annotated ground-truth region boxes, evaluating semantic grounding precision.

Input / output format

Input: Image or video frames (8 frames at 224px resolution for video) paired with a text prompt/question.

Output: Text response generated by the VLM; evaluated via exact match/accuracy for QA datasets or LLM-assistant scoring for open-ended benchmarks.

Scoring recipe

def compute_metrics(predictions, gold_answers, token_scores=None, gt_boxes=None):
    acc = sum(1 for p, g in zip(predictions, gold_answers) if is_correct(p, g)) / len(predictions)
    metrics = {'accuracy': acc}
    if token_scores is not None:
        metrics['score_map_entropy'] = -sum(s * log(s) for s in token_scores)
    if gt_boxes is not None:
        metrics['token_box_iou'] = compute_iou(selected_tokens, gt_boxes)
        metrics['attention_distance'] = compute_spatial_deviation(selected_tokens, gt_boxes)
    return metrics

Common pitfalls

• Baseline performance numbers are directly copied from the VisPruner paper rather than re-run under identical configurations, which may introduce unfair comparison artifacts.
• Token budgets are reported as absolute counts (e.g., 128, 64, 32 for images; 455, 227, 114 for video) rather than fixed pruning ratios, making cross-model compression comparisons sensitive to base token counts.
• The 'beyond-lossless' accuracy gains (>100% relative to upper bound) stem from filtering noisy tokens rather than actual model improvement, which can mislead readers expecting monotonic degradation with pruning.

Evidence (verbatim from paper)

FlashVLM achieves an average accuracy of 100.60%, surpassing even the reported performance of the unpruned (upper-bound) model.

Citation

@misc{cai2025flashvlm,
  title={FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2512.20561}
}

• arXiv: 2512.20561