# Sage Long Video Agents

> Enable agents to reason over long videos through multi-turn reasoning for complex questions and single-turn answering for simpler queries. Equip orchestrator VLM with multiple tools (web search, speech transcription, temporal grounding). Use synthetic data generation and multi-reward RL achieving 6.1% improvements on open-ended tasks.

- Skill: `adu2021/sage-long-video-agents` (Agent Skill)
- Install (CLI): `npx skillmds@latest add adu2021/sage-long-video-agents`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/sage-long-video-agents/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: adu2021 (https://skillmd.com/u/adu2021)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adu2021/sage-long-video-agents

---


## Skill Summary

SAGE introduces an agent-based system for long video reasoning operating in two paradigms: multi-turn reasoning for complex questions and single-turn answering for simpler queries. The orchestrator VLM (SAGE-MM) is equipped with multiple tools including web search, speech transcription, and temporal grounding. Synthetic data generation leverages Gemini-2.5-Flash achieving nearly 100× cost and 10× time savings. Multi-reward RL approach combines step-level rewards with LLM-as-judge evaluation, achieving 6.1% improvements on open-ended tasks and 8.2% gains for videos exceeding 10 minutes.

## When To Use

- Building systems for long-duration video understanding and reasoning
- Scenarios requiring adaptive reasoning strategies based on question complexity
- Projects where temporal grounding and external tools enhance reasoning
- Research on multi-turn video agents with RL optimization

## When NOT To Use

- Short-form video tasks not benefiting from multi-turn complexity
- Applications without access to external tools or context-augmentation
- Scenarios where synthetic data quality is problematic
- Domains where simpler single-pass models suffice

## Core Technique

Multiple innovations enable effective long video reasoning:

**1. System Design**
Use orchestrator VLM (SAGE-MM) equipped with multiple tools rather than relying solely on temporal grounding:
- Web search: external context
- Speech transcription: audio understanding
- Temporal grounding: video-specific localization

Implement two paradigms:
- Multi-turn reasoning: complex questions requiring multiple reasoning steps
- Single-turn answering: simpler queries with direct answers

**2. Synthetic Data Generation**
Leverage Gemini-2.5-Flash to generate high-quality QnA pairs efficiently, achieving "nearly 100× cost and 10× time savings compared to human annotation and subclip processing pipelines."

**3. RL Post-Training Recipe**
Implement multi-reward approach combining:
- Step-level rewards: format, reasonable tool use, argument validity
- LLM-as-judge evaluation: for open-ended problems avoiding string-matching metrics

**4. SAGE-Bench Evaluation**
Curated benchmark with "1744 manually verified samples spanning diverse durations" emphasizing open-ended questions from entertainment videos.

## Results

- Open-ended tasks: 6.1% improvement
- Videos exceeding 10 minutes: 8.2% improvement
- Validates effectiveness of adaptive reasoning for variable-duration content

## Implementation Notes

Design orchestrator VLM with tool integration. Implement multi-turn vs. single-turn reasoning routing. Use synthetic data generation for training pairs. Implement multi-reward RL with step-level and judge-based evaluation. Evaluate on videos of varying durations and question complexities.

## References

- Original paper: SAGE (Dec 2025)
- Agent-based video understanding
- Long-context video reasoning