qhjqhj00 streammecoeval
StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding — Wang et al. (2026) (arXiv:2604.09000, 2026)
What this evaluates
Evaluates the ability of long-term agent memory compression methods to retain and retrieve critical information from streaming and long-form videos. It probes how effectively a model can maintain temporal consistency and answer complex queries while significantly reducing memory graph size, measured via answer accuracy across robotic, web, and general video benchmarks.
Datasets
- M3-Bench-robot — total ?; splits: test (-1)
- M3-Bench-web — total ?; splits: test (-1)
- Video-MME-Long — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered video-based queries, evaluated by GPT-4o. Reported per category (ME, MH, CM, PU, GK) and averaged across datasets.
Input / output format
Input: Video sequences (streaming or offline) paired with natural language queries/questions.
Output: Natural language answers generated by the model, which are then scored by GPT-4o for correctness.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
if gpt4o_judge(pred, gold) == 'correct':
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Using GPT-4o as the sole judge for answer quality may introduce LLM-as-a-judge bias or inconsistency across different video domains.
- Comparing compression performance across datasets with different inherent redundancies (e.g., robotic vs. YouTube) without normalizing for baseline accuracy differences.
- Assuming streaming benchmarks behave identically to offline video benchmarks, ignoring temporal continuity and retrieval latency constraints.
Evidence (verbatim from paper)
Consistent with the original M3-Agent setup, we use GPT-4o to assess the answer quality and employ text-embedding-3-large to encode the query content generated by M3-Agent to supporting subsequent memory graph retrieval. ... Even with 70% of text nodes compressed, StreamMeCo and TMR mechanism still achieves an average accuracy improvement of 1.0% across all datasets compared to the uncompressed M3-Agent.
Citation
@misc{wang2026streammecol,
title={StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding},
author={Wang et al. (2026)},
year={2026},
note={arXiv:2604.09000}
}
- arXiv: 2604.09000