Video Detective

Clue Hunting Mechanism

VideoDetective decomposes long-video understanding into three iterative phases:

Phase 1: Hypothesis

Select candidate video segments based on initial query relevance estimation. Rather than exhaustive sampling, prioritize segments likely to contain answer evidence.

Phase 2: Verification

Extract multimodal evidence from selected segments:

• Visual: Frame content and appearance
• Captions: Automatically generated scene descriptions
• OCR: On-screen text (subtitles, labels, overlays)
• Speech: Transcript of audio content

Score each segment's relevance to query using evidence fusion.

Phase 3: Refinement

Propagate relevance scores across visual-temporal affinity graph to estimate unobserved segments, creating global belief field over entire video.

This sparse observation approach requires far fewer tokens than methods that densely sample and process every frame.

Extrinsic vs. Intrinsic Relevance

Extrinsic Relevance

Query-to-segment direct matching: "How well does this segment answer the user's question?"

• Based on multimodal evidence (captions, OCR, speech)
• Direct semantic alignment with query
• Observable only by processing the segment

Intrinsic Relevance

Video internal structure: "What's the semantic relationship between segments?"

• Segments clustered together likely share meaning
• Temporal proximity correlates with semantic continuity
• Visual similarity suggests related content
• Inferred without processing every segment

Joint Modeling: Combine both signals: relevance = extrinsic confidence × intrinsic structure propagation

This enables the system to say "we only observed segment 5, which was relevant, and segments 4 and 6 are visually similar and adjacent, so they're likely relevant too."

Frame Identification Strategy

Rather than pixel-level frame selection, VideoDetective operates at segment granularity:

Graph Construction: Build visual-temporal affinity graph encoding:

• Visual similarity between segment pairs (CNN feature distance)
• Temporal proximity (segments close in time connected strongly)
• Result: k-nearest neighbor graph on segment embeddings

Selective Observation: Use active learning to choose anchor segments:

1. High uncertainty regions (marginal relevance)
2. Representative of different video regions
3. Maximize information gain per token

Propagation via Graph Diffusion: For each observed anchor segment:

1. Extract multimodal evidence (captions, OCR, speech)
2. Compute extrinsic relevance score
3. Run graph diffusion to propagate to unobserved neighbors
4. Create global belief field over all segments

Diversity via Graph-NMS: Apply Non-Maximum Suppression on graph:

• Select high-confidence segments
• Suppress nearby neighbors (avoid redundancy)
• Ensures semantic aspect coverage across video

Performance Characteristics

Token Efficiency

Achieves competitive accuracy with significantly fewer visual tokens than:

• Dense frame sampling (every N frames)
• Uniform segment coverage (cover entire video equally)
• Multi-scale sampling (process at different resolutions)

Exact speedup depends on:

• Video length (longer videos benefit more from sparsity)
• Query complexity (localized queries use fewer segments)
• Segment-to-token ratio (coarse segments require fewer tokens)

Accuracy-Efficiency Tradeoff

• More observation budget → higher accuracy, more tokens
• Fewer observations → faster inference, potential accuracy loss
• Graph propagation acts as implicit regularizer (neighboring segments should have similar relevance)

Conditions of Applicability

Works well when:

• Video has clear temporal structure (coherent scenes, semantic continuity)
• Query is localized (answer in specific clips, not scattered throughout)
• Visual similarity correlates with semantic similarity (content is organized spatially)
• Token budget is limited (sparse sampling critical for efficiency)

Less optimal when:

• Query requires integrating evidence across entire video (non-localized)
• Video structure is fragmented (random cuts, rapid scene changes)
• Visual similarity misleads (visually similar scenes have different meanings)
• Dense processing is needed (safety-critical applications requiring comprehensive coverage)

Integration Pattern

Input Assumptions:

• Segmented video (pre-computed clip boundaries)
• Query in natural language
• Optional: pre-computed segment embeddings/captions

Output:

• Ranked list of relevant segments
• Confidence scores (from relevance propagation)
• Evidence excerpts (captions, OCR, speech snippets)

Minimal Wrapper:

Long video query
    ↓
Segment hypothesis (initial ranking)
    ↓
Selective observation + evidence extraction
    ↓
Graph diffusion (propagate to unobserved)
    ↓
Graph-NMS (diversify + select top-K)
    ↓
Relevant clips + evidence

Works with any VLM capable of multimodal understanding without requiring architectural changes.