Hermes Hierarchical Video Memory

HERMES: Hierarchical Video Memory via KV Cache

This skill demonstrates how to leverage KV cache as hierarchical memory for efficient streaming video understanding, enabling real-time inference with significantly reduced computational and memory overhead.

When to Use

• Real-time video stream processing (surveillance, robotics, autonomous vehicles)
• Continuous video understanding with strict latency requirements
• Systems with limited GPU memory (mobile, edge devices)
• Applications requiring fast temporal reasoning over video
• Scenarios where 10x speedup in response time is valuable

When NOT to Use

• Offline batch video analysis (efficiency gains matter less)
• Short video clips (hierarchical memory overhead not justified)
• Single-frame analysis (inherently doesn't need temporal memory)
• Systems with unlimited compute resources and no latency constraints

Key Concept

Standard video transformers process entire video sequences, creating bottlenecks from expensive KV cache computations. HERMES restructures the KV cache as a hierarchical memory:

1. Frame-Level Cache: Store KV for individual frames
2. Temporal Compression: Compress older frames into summary representations
3. Hierarchical Queries: Efficient retrieval across time scales
4. Streaming Updates: Incrementally add new frames without recomputing entire history

This maintains temporal understanding while staying memory-efficient.

Implementation Pattern

Structure KV cache hierarchically for streaming video:

# Pseudocode for hierarchical KV cache management
class HierarchicalKVCache:
    def __init__(self, cache_levels=3, compression_ratio=4):
        self.cache_levels = cache_levels  # Multiple time scales
        self.compression_ratio = compression_ratio
        self.caches = [[] for _ in range(cache_levels)]

    def process_frame(self, frame, frame_idx):
        # Compute KV for current frame
        frame_k, frame_v = self.compute_kv(frame)

        # Store at finest granularity
        self.caches[0].append((frame_k, frame_v))

        # Hierarchical compression: aggregate into coarser levels
        if frame_idx % self.compression_ratio == 0:
            self.compress_to_next_level(from_level=0, to_level=1)

        if frame_idx % (self.compression_ratio ** 2) == 0:
            self.compress_to_next_level(from_level=1, to_level=2)

        return frame_k, frame_v

    def compress_to_next_level(self, from_level, to_level):
        # Aggregate N recent frames into compressed representation
        frames_to_compress = self.caches[from_level][-self.compression_ratio:]

        # Summarize: combine KV through pooling/attention
        compressed_k = pool_keys(frames_to_compress)
        compressed_v = pool_values(frames_to_compress)

        self.caches[to_level].append((compressed_k, compressed_v))

    def get_context_for_query(self, query_level=0):
        # Gather KV from all hierarchical levels
        context = []
        context.extend(self.caches[0])  # Fine detail
        if len(self.caches[1]) > 0:
            context.extend(self.caches[1])  # Medium-term patterns
        if len(self.caches[2]) > 0:
            context.extend(self.caches[2])  # Long-term context

        return context

The hierarchy enables efficient querying: recent frames in detail, older content in compressed form.

Key Results

• 10x faster response times compared to standard video transformers
• Minimal GPU memory overhead despite processing long video sequences
• Maintains understanding of temporal patterns across multiple scales
• Training-free: applies to existing video understanding models

Research Context

This work shows that KV cache, typically seen as a computational bottleneck, can be restructured as a feature—a hierarchical memory system that balances recency with efficiency. By organizing temporal information across scales, video understanding becomes feasible in real-time applications.