Sage Long Video Agents

Skill Summary

SAGE introduces an agent-based system for long video reasoning operating in two paradigms: multi-turn reasoning for complex questions and single-turn answering for simpler queries. The orchestrator VLM (SAGE-MM) is equipped with multiple tools including web search, speech transcription, and temporal grounding. Synthetic data generation leverages Gemini-2.5-Flash achieving nearly 100× cost and 10× time savings. Multi-reward RL approach combines step-level rewards with LLM-as-judge evaluation, achieving 6.1% improvements on open-ended tasks and 8.2% gains for videos exceeding 10 minutes.

When To Use

• Building systems for long-duration video understanding and reasoning
• Scenarios requiring adaptive reasoning strategies based on question complexity
• Projects where temporal grounding and external tools enhance reasoning
• Research on multi-turn video agents with RL optimization

When NOT To Use

• Short-form video tasks not benefiting from multi-turn complexity
• Applications without access to external tools or context-augmentation
• Scenarios where synthetic data quality is problematic
• Domains where simpler single-pass models suffice

Core Technique

Multiple innovations enable effective long video reasoning:

1. System Design Use orchestrator VLM (SAGE-MM) equipped with multiple tools rather than relying solely on temporal grounding:

• Web search: external context
• Speech transcription: audio understanding
• Temporal grounding: video-specific localization

Implement two paradigms:

• Multi-turn reasoning: complex questions requiring multiple reasoning steps
• Single-turn answering: simpler queries with direct answers

2. Synthetic Data Generation Leverage Gemini-2.5-Flash to generate high-quality QnA pairs efficiently, achieving "nearly 100× cost and 10× time savings compared to human annotation and subclip processing pipelines."

3. RL Post-Training Recipe Implement multi-reward approach combining:

• Step-level rewards: format, reasonable tool use, argument validity
• LLM-as-judge evaluation: for open-ended problems avoiding string-matching metrics

4. SAGE-Bench Evaluation Curated benchmark with "1744 manually verified samples spanning diverse durations" emphasizing open-ended questions from entertainment videos.

Results

• Open-ended tasks: 6.1% improvement
• Videos exceeding 10 minutes: 8.2% improvement
• Validates effectiveness of adaptive reasoning for variable-duration content

Implementation Notes

Design orchestrator VLM with tool integration. Implement multi-turn vs. single-turn reasoning routing. Use synthetic data generation for training pairs. Implement multi-reward RL with step-level and judge-based evaluation. Evaluate on videos of varying durations and question complexities.

References

• Original paper: SAGE (Dec 2025)
• Agent-based video understanding
• Long-context video reasoning