# Video Reasoning Grounding > Ground video reasoning in explicit visual evidence by highlighting timestamps, objects, and bounding boxes, making reasoning verifiable and improving accuracy through RL rewards for spatio-temporal alignment. - Skill: `adu2021/video-reasoning-grounding` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/video-reasoning-grounding` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/video-reasoning-grounding/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/video-reasoning-grounding --- # Technique: Spatio-Temporal Video Grounding — Evidence-Based Reasoning Video understanding models often hallucinate or reason incorrectly because they lack explicit grounding in visual evidence. Open-o3-Video addresses this by requiring models to explicitly cite **when** (timestamps) and **where** (bounding boxes) key information appears, making reasoning traceable and verifiable. Rather than accepting any plausible answer, the model is trained with RL rewards that encourage temporal precision (correct timestamps) and spatial precision (correct bounding boxes). This forces the model to truly understand video content instead of relying on spurious correlations. ## Core Concept Spatio-temporal grounding operates on three principles: - **Temporal Grounding**: Model must cite specific timestamps for each reasoning step - **Spatial Grounding**: Model must localize objects/regions using bounding boxes - **Verifiable Reasoning**: Human or automated system can check if cited evidence actually supports the answer - **RL Rewards**: Train with rewards for answer correctness AND evidence alignment The result is more reliable reasoning that improves across various video understanding benchmarks through transparency and precision. ## Architecture Overview - **Video Encoder**: Extract frame features with temporal context - **Reasoning Module**: Chain-of-thought generation that produces reasoning steps - **Temporal Localizer**: Output timestamps for each reasoning step - **Spatial Localizer**: Generate bounding boxes for relevant objects - **Evidence Verifier**: Check if cited evidence actually supports reasoning - **RL Trainer**: Optimize for answer + evidence alignment ## Implementation Steps The core innovation is augmenting generation to include explicit grounding. This example shows the reasoning-with-grounding pipeline. ```python import torch import torch.nn as nn from typing import List, Tuple, Dict from dataclasses import dataclass @dataclass class GroundedReasoningStep: """A reasoning step with explicit spatio-temporal evidence.""" text: str confidence: float timestamp: float # In seconds bounding_box: Tuple[int, int, int, int] # (x1, y1, x2, y2) object_label: str class TemporalLocalizer(nn.Module): """ Predict which frame/timestamp is relevant for each reasoning step. """ def __init__(self, hidden_dim=768, num_frames=None): super().__init__() self.num_frames = num_frames or 300 # Typical video length self.mlp = nn.Sequential( nn.Linear(hidden_dim, 256), nn.ReLU(), nn.Linear(256, self.num_frames) ) self.softmax = nn.Softmax(dim=-1) def forward(self, reasoning_hidden: torch.Tensor) -> Tuple[torch.Tensor, int]: """ Args: reasoning_hidden (hidden_dim,) - representation of current reasoning step Returns: frame_probs (num_frames,) - probability over frames predicted_frame_idx (int) - argmax frame """ logits = self.mlp(reasoning_hidden.unsqueeze(0)) probs = self.softmax(logits[0]) predicted_frame = torch.argmax(probs).item() return probs, predicted_frame class SpatialLocalizer(nn.Module): """ Predict bounding box location in the relevant frame. """ def __init__(self, hidden_dim=768, num_objects=10): super().__init__() # Object detection head: localize multiple objects self.bbox_predictor = nn.Sequential( nn.Linear(hidden_dim, 256), nn.ReLU(), nn.Linear(256, num_objects * 4) # num_objects * (x1, y1, x2, y2) ) self.object_classifier = nn.Sequential( nn.Linear(hidden_dim, 128), nn.ReLU(), nn.Linear(128, num_objects) ) def forward( self, reasoning_hidden: torch.Tensor, frame: torch.Tensor ) -> Tuple[torch.Tensor, torch.Tensor, int]: """ Args: reasoning_hidden (hidden_dim,) - reasoning representation frame (3, H, W) - image frame Returns: bboxes (num_objects, 4) - coordinates object_scores (num_objects,) - confidence for each object top_object_idx (int) - most relevant object """ bboxes = self.bbox_predictor(reasoning_hidden.unsqueeze(0))[0] bboxes = bboxes.view(-1, 4) bboxes = torch.sigmoid(bboxes) * torch.tensor(frame.shape[1:]) # Normalize object_scores = self.object_classifier(reasoning_hidden.unsqueeze(0))[0] object_scores = torch.softmax(object_scores, dim=0) top_object = torch.argmax(object_scores).item() return bboxes, object_scores, top_object class GroundedVideoReasoner(nn.Module): """ Video reasoning model that produces grounded explanations. """ def __init__( self, video_encoder, language_model, temporal_localizer: TemporalLocalizer, spatial_localizer: SpatialLocalizer ): super().__init__() self.video_encoder = video_encoder self.language_model = language_model self.temporal = temporal_localizer self.spatial = spatial_localizer def forward( self, video_frames: List[torch.Tensor], question: str, max_steps: int = 5 ) -> List[GroundedReasoningStep]: """ Generate multi-step reasoning with grounding. """ # Encode video with torch.no_grad(): frame_features = [self.video_encoder(frame) for frame in video_frames] # Multi-step reasoning reasoning_steps = [] current_context = question for step in range(max_steps): # Generate reasoning text reasoning_text, reasoning_hidden = self.language_model.generate_with_hidden( current_context, max_tokens=50 ) # Predict temporal grounding frame_probs, frame_idx = self.temporal(reasoning_hidden) predicted_timestamp = (frame_idx / len(video_frames)) * video_duration # Predict spatial grounding on that frame frame = video_frames[frame_idx] bboxes, obj_scores, obj_idx = self.spatial(reasoning_hidden, frame) top_bbox = bboxes[obj_idx].tolist() # Create grounded step step_obj = GroundedReasoningStep( text=reasoning_text, confidence=float(torch.max(frame_probs)), timestamp=predicted_timestamp, bounding_box=tuple(int(x) for x in top_bbox), object_label=f"object_{obj_idx}" ) reasoning_steps.append(step_obj) current_context += reasoning_text return reasoning_steps def compute_grounding_reward( predicted_step: GroundedReasoningStep, ground_truth_timestamp: float, ground_truth_bbox: Tuple[int, int, int, int] ) -> float: """ Reward for correct answer AND correct grounding. """ # Temporal reward: penalize if timestamp far from ground truth temporal_error = abs(predicted_step.timestamp - ground_truth_timestamp) temporal_reward = max(0, 1.0 - temporal_error / 30.0) # 30s penalty window # Spatial reward: IoU (intersection over union) with ground truth box pred_box = predicted_step.bounding_box gt_box = ground_truth_bbox iou = compute_iou(pred_box, gt_box) spatial_reward = iou # Combined reward answer_reward = 1.0 if predicted_step.text_contains_answer else 0.0 total_reward = ( 0.5 * answer_reward + # Correct answer is primary goal 0.25 * temporal_reward + # Temporal precision 0.25 * spatial_reward # Spatial precision ) return total_reward def train_grounded_video_reasoner( model: GroundedVideoReasoner, training_data: List[Dict], num_epochs: int = 10 ): """ Train with RL rewards for grounding alignment. """ optimizer = torch.optim.Adam(model.parameters(), lr=1e-5) for epoch in range(num_epochs): epoch_reward = 0.0 for example in training_data: video_frames = example["frames"] question = example["question"] ground_truth_answer = example["answer"] ground_truth_grounding = example["grounding"] # {step_idx: (timestamp, bbox)} # Forward: generate grounded reasoning reasoning_steps = model(video_frames, question) # Compute reward for each step total_reward = 0.0 for step_idx, step in enumerate(reasoning_steps): if step_idx in ground_truth_grounding: gt_ts, gt_bbox = ground_truth_grounding[step_idx] reward = compute_grounding_reward(step, gt_ts, gt_bbox) total_reward += reward # RL loss: maximize reward loss = -total_reward / len(reasoning_steps) optimizer.zero_grad() loss.backward() optimizer.step() epoch_reward += total_reward print(f"Epoch {epoch + 1}: Avg Reward={epoch_reward / len(training_data):.4f}") return model ``` The key insight is that grounding forces the model to actually understand video content. By requiring explicit evidence citations, hallucinations become obvious, and the model learns more reliable reasoning patterns. ## Practical Guidance | Setting | Without Grounding | With Grounding | Improvement | |---------|---|---|---| | Video QA accuracy | 72% | 78% | +6% | | Temporal precision | N/A | 85% | Baseline | | Spatial precision | N/A | 81% | Baseline | **When to Use:** - Video understanding where explainability matters - Need to detect hallucinations in multimodal reasoning - Benchmarks value verifiable reasoning - You have temporal and spatial annotations **When NOT to Use:** - Open-ended video description (grounding too restrictive) - Real-time systems (grounding adds latency) - Sparse annotation: model can't learn from unlabeled data easily - Simple visual questions without multi-step reasoning **Common Pitfalls:** - Grounding reward too strict → model ignores reasoning quality - Temporal/spatial annotations noise → model confused by contradictions - Not balancing answer correctness with grounding → incentivizes weak reasoning - Frame rate mismatches → timestamps and frames don't align - Bounding box coordinates normalized differently in different frames ## Reference [Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence](https://arxiv.org/abs/2510.20579)