Flashcache Eval

flashcache-eval

Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach — Yang et al. (2025) (arXiv:2511.16786, 2025)

What this evaluates

Evaluates the effectiveness of a frequency-domain-guided KV cache compression method (FlashCache) on multimodal long-context understanding tasks. It measures how well the model preserves accuracy under varying KV cache retention ratios and quantifies the computational overhead and decoding latency compared to baseline eviction methods.

Datasets

• MileBench — total ?; splits: test (-1)
• MUIRBench — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• V* — total ?; splits: test (-1)
• HR-Bench — total ?; splits: test (-1)
• FAVOR-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions or tasks. Computed as (number of correct predictions / total number of instances) * 100.
• decoding_latency — range: ms
  • Time in milliseconds to generate a 100-token output sequence, measured across varying input lengths (2K to 64K tokens).
• method_overhead — range: ms
  • Additional computational time in milliseconds introduced by the KV cache eviction/compression method itself, measured for input lengths 2K, 4K, and 8K.

Input / output format

Input: Multimodal prompts (images, videos, text) processed by MLLMs (LLaVA-OneVision-1.5-8B-Instruct, Qwen2.5-VL-7B/32B-Instruct) with KV cache retention ratio $\rho$ applied.

Output: Generated text responses evaluated against ground truth labels for accuracy.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• KV retention ratio $\rho$ is a critical hyperparameter; performance degrades sharply below 0.1, so results must be reported at specific ratios (0.2, 0.1, 0.05).
• Several baselines (H2O, LOOK-M, MEDA) encounter Out-Of-Memory (OOM) errors on long-context benchmarks, preventing direct comparison at lower retention ratios.
• HR-Bench accuracy is evaluated using a separate LLM judge (Qwen2-32B), which may introduce judge bias compared to exact-match metrics.

Evidence (verbatim from paper)

We employ Qwen2.5-VL-7B for the experiment with retention ratios of 0.1 and 0.05 and present the accuracy metrics for different compression methods.

Citation

@misc{yang2025flashcache,
  title={Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.16786}
}

• arXiv: 2511.16786