Video Reasoning Grounding

Technique: Spatio-Temporal Video Grounding — Evidence-Based Reasoning

Video understanding models often hallucinate or reason incorrectly because they lack explicit grounding in visual evidence. Open-o3-Video addresses this by requiring models to explicitly cite when (timestamps) and where (bounding boxes) key information appears, making reasoning traceable and verifiable.

Rather than accepting any plausible answer, the model is trained with RL rewards that encourage temporal precision (correct timestamps) and spatial precision (correct bounding boxes). This forces the model to truly understand video content instead of relying on spurious correlations.

Core Concept

Spatio-temporal grounding operates on three principles:

• Temporal Grounding: Model must cite specific timestamps for each reasoning step
• Spatial Grounding: Model must localize objects/regions using bounding boxes
• Verifiable Reasoning: Human or automated system can check if cited evidence actually supports the answer
• RL Rewards: Train with rewards for answer correctness AND evidence alignment

The result is more reliable reasoning that improves across various video understanding benchmarks through transparency and precision.

Architecture Overview

• Video Encoder: Extract frame features with temporal context
• Reasoning Module: Chain-of-thought generation that produces reasoning steps
• Temporal Localizer: Output timestamps for each reasoning step
• Spatial Localizer: Generate bounding boxes for relevant objects
• Evidence Verifier: Check if cited evidence actually supports reasoning
• RL Trainer: Optimize for answer + evidence alignment

Implementation Steps

The core innovation is augmenting generation to include explicit grounding. This example shows the reasoning-with-grounding pipeline.

import torch
import torch.nn as nn
from typing import List, Tuple, Dict
from dataclasses import dataclass

@dataclass
class GroundedReasoningStep:
    """A reasoning step with explicit spatio-temporal evidence."""
    text: str
    confidence: float
    timestamp: float  # In seconds
    bounding_box: Tuple[int, int, int, int]  # (x1, y1, x2, y2)
    object_label: str


class TemporalLocalizer(nn.Module):
    """
    Predict which frame/timestamp is relevant for each reasoning step.
    """

    def __init__(self, hidden_dim=768, num_frames=None):
        super().__init__()
        self.num_frames = num_frames or 300  # Typical video length
        self.mlp = nn.Sequential(
            nn.Linear(hidden_dim, 256),
            nn.ReLU(),
            nn.Linear(256, self.num_frames)
        )
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, reasoning_hidden: torch.Tensor) -> Tuple[torch.Tensor, int]:
        """
        Args: reasoning_hidden (hidden_dim,) - representation of current reasoning step
        Returns:
            frame_probs (num_frames,) - probability over frames
            predicted_frame_idx (int) - argmax frame
        """
        logits = self.mlp(reasoning_hidden.unsqueeze(0))
        probs = self.softmax(logits[0])
        predicted_frame = torch.argmax(probs).item()
        return probs, predicted_frame


class SpatialLocalizer(nn.Module):
    """
    Predict bounding box location in the relevant frame.
    """

    def __init__(self, hidden_dim=768, num_objects=10):
        super().__init__()
        # Object detection head: localize multiple objects
        self.bbox_predictor = nn.Sequential(
            nn.Linear(hidden_dim, 256),
            nn.ReLU(),
            nn.Linear(256, num_objects * 4)  # num_objects * (x1, y1, x2, y2)
        )
        self.object_classifier = nn.Sequential(
            nn.Linear(hidden_dim, 128),
            nn.ReLU(),
            nn.Linear(128, num_objects)
        )

    def forward(
        self,
        reasoning_hidden: torch.Tensor,
        frame: torch.Tensor
    ) -> Tuple[torch.Tensor, torch.Tensor, int]:
        """
        Args:
            reasoning_hidden (hidden_dim,) - reasoning representation
            frame (3, H, W) - image frame
        Returns:
            bboxes (num_objects, 4) - coordinates
            object_scores (num_objects,) - confidence for each object
            top_object_idx (int) - most relevant object
        """
        bboxes = self.bbox_predictor(reasoning_hidden.unsqueeze(0))[0]
        bboxes = bboxes.view(-1, 4)
        bboxes = torch.sigmoid(bboxes) * torch.tensor(frame.shape[1:])  # Normalize

        object_scores = self.object_classifier(reasoning_hidden.unsqueeze(0))[0]
        object_scores = torch.softmax(object_scores, dim=0)

        top_object = torch.argmax(object_scores).item()
        return bboxes, object_scores, top_object


class GroundedVideoReasoner(nn.Module):
    """
    Video reasoning model that produces grounded explanations.
    """

    def __init__(
        self,
        video_encoder,
        language_model,
        temporal_localizer: TemporalLocalizer,
        spatial_localizer: SpatialLocalizer
    ):
        super().__init__()
        self.video_encoder = video_encoder
        self.language_model = language_model
        self.temporal = temporal_localizer
        self.spatial = spatial_localizer

    def forward(
        self,
        video_frames: List[torch.Tensor],
        question: str,
        max_steps: int = 5
    ) -> List[GroundedReasoningStep]:
        """
        Generate multi-step reasoning with grounding.
        """
        # Encode video
        with torch.no_grad():
            frame_features = [self.video_encoder(frame) for frame in video_frames]

        # Multi-step reasoning
        reasoning_steps = []
        current_context = question

        for step in range(max_steps):
            # Generate reasoning text
            reasoning_text, reasoning_hidden = self.language_model.generate_with_hidden(
                current_context,
                max_tokens=50
            )

            # Predict temporal grounding
            frame_probs, frame_idx = self.temporal(reasoning_hidden)
            predicted_timestamp = (frame_idx / len(video_frames)) * video_duration

            # Predict spatial grounding on that frame
            frame = video_frames[frame_idx]
            bboxes, obj_scores, obj_idx = self.spatial(reasoning_hidden, frame)
            top_bbox = bboxes[obj_idx].tolist()

            # Create grounded step
            step_obj = GroundedReasoningStep(
                text=reasoning_text,
                confidence=float(torch.max(frame_probs)),
                timestamp=predicted_timestamp,
                bounding_box=tuple(int(x) for x in top_bbox),
                object_label=f"object_{obj_idx}"
            )

            reasoning_steps.append(step_obj)
            current_context += reasoning_text

        return reasoning_steps


def compute_grounding_reward(
    predicted_step: GroundedReasoningStep,
    ground_truth_timestamp: float,
    ground_truth_bbox: Tuple[int, int, int, int]
) -> float:
    """
    Reward for correct answer AND correct grounding.
    """
    # Temporal reward: penalize if timestamp far from ground truth
    temporal_error = abs(predicted_step.timestamp - ground_truth_timestamp)
    temporal_reward = max(0, 1.0 - temporal_error / 30.0)  # 30s penalty window

    # Spatial reward: IoU (intersection over union) with ground truth box
    pred_box = predicted_step.bounding_box
    gt_box = ground_truth_bbox

    iou = compute_iou(pred_box, gt_box)
    spatial_reward = iou

    # Combined reward
    answer_reward = 1.0 if predicted_step.text_contains_answer else 0.0

    total_reward = (
        0.5 * answer_reward +  # Correct answer is primary goal
        0.25 * temporal_reward +  # Temporal precision
        0.25 * spatial_reward  # Spatial precision
    )

    return total_reward


def train_grounded_video_reasoner(
    model: GroundedVideoReasoner,
    training_data: List[Dict],
    num_epochs: int = 10
):
    """
    Train with RL rewards for grounding alignment.
    """
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

    for epoch in range(num_epochs):
        epoch_reward = 0.0

        for example in training_data:
            video_frames = example["frames"]
            question = example["question"]
            ground_truth_answer = example["answer"]
            ground_truth_grounding = example["grounding"]  # {step_idx: (timestamp, bbox)}

            # Forward: generate grounded reasoning
            reasoning_steps = model(video_frames, question)

            # Compute reward for each step
            total_reward = 0.0
            for step_idx, step in enumerate(reasoning_steps):
                if step_idx in ground_truth_grounding:
                    gt_ts, gt_bbox = ground_truth_grounding[step_idx]
                    reward = compute_grounding_reward(step, gt_ts, gt_bbox)
                    total_reward += reward

            # RL loss: maximize reward
            loss = -total_reward / len(reasoning_steps)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            epoch_reward += total_reward

        print(f"Epoch {epoch + 1}: Avg Reward={epoch_reward / len(training_data):.4f}")

    return model

The key insight is that grounding forces the model to actually understand video content. By requiring explicit evidence citations, hallucinations become obvious, and the model learns more reliable reasoning patterns.

Practical Guidance

When to Use:

• Video understanding where explainability matters
• Need to detect hallucinations in multimodal reasoning
• Benchmarks value verifiable reasoning
• You have temporal and spatial annotations

When NOT to Use:

• Open-ended video description (grounding too restrictive)
• Real-time systems (grounding adds latency)
• Sparse annotation: model can't learn from unlabeled data easily
• Simple visual questions without multi-step reasoning

Common Pitfalls:

• Grounding reward too strict → model ignores reasoning quality
• Temporal/spatial annotations noise → model confused by contradictions
• Not balancing answer correctness with grounding → incentivizes weak reasoning
• Frame rate mismatches → timestamps and frames don't align
• Bounding box coordinates normalized differently in different frames

Reference

Open-o3 Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence