dingxingdi Skill: video-text temporal grounding
1. Capability Definition & Real Case
- Professional Definition: The ability to accurately localize and retrieve specific temporal intervals (start and end timestamps) within a continuous video stream that correspond to a natural language description. This requires the agent to align sequential visual frames with semantic concepts, maintain temporal coherence over sampled frames, and differentiate between adjacent or similar actions within a complex visual environment.
- Dimension Hierarchy: Specialized Knowledge Navigation -> Corpus-Specific Retrieval & Reasoning -> video-text temporal grounding
Real Case
[Case 1]
- Initial Environment: A RAG environment containing a news broadcast video as the primary retrieved asset. The video is represented as a sequence of dynamic resolution frames stored in a multimodal database.
- Real Question: Select the clips of the video that contain the reporter discussing the specific details of the rescue mission mission.
- Real Trajectory: The agent samples 8 frames uniformly across the video to establish a coarse timeline. It identifies high-resolution visual landmarks of a rescue boat at the 45-second mark. Using the temporal rotary position embedding (M-RoPE), it inspects adjacent frames to confirm the exact moment the discussion starts and ends.
- Real Answer: 00:45 - 01:22
- Why this demonstrates the capability: This case demonstrates temporal grounding because the answer is not a single fact, but a duration. The agent must successfully navigate the visual sequence to define boundaries that satisfy the semantic constraints of the query.
[Case 2]
- Initial Environment: A multimodal corpus of instructional cooking videos (simulated from YouCook2/Breakfast datasets). Each video contains multiple discrete steps in a complex recipe.
- Real Question: At what point in the video does the chef start adding the milk to the batter and when is the pouring finished?
- Real Trajectory: The agent retrieves the 'Pancake Preparation' video. It performs iterative temporal narrowing, starting with 3D-convolution based motion detection to find the 'pouring' action and then zooming in on frames to identify the white liquid entity vs. the bowl batter.
- Real Answer: 03:12 - 03:28
- Why this demonstrates the capability: Grounding requires distinguishing between 'preparing' to pour and the 'actual' pouring. The agent's ability to provide a precise [Start, End] interval shows it can resolve actions into structured temporal facts.
Pipeline Execution Instructions
To synthesize data for this capability, you must strictly follow a 3-phase pipeline. Do not hallucinate steps. Read the corresponding reference file for each phase sequentially:
Phase 1: Environment Exploration Read the exploration guidelines to discover raw knowledge seeds:
references/EXPLORATION.mdPhase 2: Trajectory Selection Once Phase 1 is complete, read the selection criteria to evaluate the trajectory:
references/SELECTION.mdPhase 3: Data Synthesis Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data:
references/SYNTHESIS.md