Hermes Video Understanding Eval

hermes-video-understanding-eval

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding — Haowei Zhang et al. (arXiv:2601.14724, 2026)

What this evaluates

Evaluates a training-free KV cache management framework for real-time streaming and offline video understanding. It probes the model's ability to maintain temporal coherence and answer questions accurately under strict token/memory budgets, while measuring inference efficiency.

Datasets

• StreamingBench — total ?; splits: test (-1)
• OVO-Bench — total ?; splits: test (-1)
• RVS (Ego & Movie) — total ?; splits: test (-1)
• MVBench — total ?; splits: test (-1)
• VideoMME — total ?; splits: test (-1)
• Egoschema — total ?; splits: dev (-1)

Metrics

• accuracy (primary) — range: [0, 100] percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.
• score — range: [1, 5] scale
  • 1-5 scale rating for open-ended answers, evaluated by GPT-3.5-turbo-0125 based on answer accuracy.
• TTFT — range: ms
  • Time to First Token, measured in milliseconds from query input to first output token generation.
• TPOT — range: ms
  • Time Per Output Token, measured in milliseconds per generated token.
• peak_gpu_memory — range: GB
  • Maximum GPU memory usage during inference, measured in gigabytes.

Input / output format

Input: Video frames processed sequentially in chunks of 16 frames, fed into the backbone LLM. Text questions provided as prompts.

Output: Text responses, either multiple-choice selections or open-ended descriptions.

Scoring recipe

def compute_metrics(predictions, golds, llm_judge='gpt-3.5-turbo-0125'):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip())
    accuracy = (correct / len(golds)) * 100
    scores = []
    for p, g in zip(predictions, golds):
        score = llm_judge.evaluate(p, g, scale=(1, 5))
        scores.append(score)
    avg_score = sum(scores) / len(scores)
    return {'accuracy': accuracy, 'avg_score': avg_score}

Common pitfalls

• Open-ended evaluation on RVS relies on GPT-3.5-turbo-0125 as a judge rather than exact string matching, which introduces LLM-judge bias and prompt sensitivity.
• Efficiency metrics (TTFT, TPOT, memory) are reported on a single A800 GPU with FP16 precision; results may vary significantly on consumer or different enterprise GPUs.
• The memory budget is fixed at 4K tokens for main results, but ablations show streaming tasks tolerate smaller budgets while long offline tasks degrade sharply below 4K.

Evidence (verbatim from paper)

On RVS-Ego and RVS-Movie (Tab. 2), we evaluate the model answer by GPT-3.5-turbo-0125 on accuracy and score (1–5 scale), consistent with compared baselines.

Citation

@misc{zhang2026hermes,
  title={HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding},
  author={Haowei Zhang et al.},
  year={2026},
  note={arXiv:2601.14724}
}

• arXiv: 2601.14724