adu2021 Title: Use Video Generation as a Universal Reasoning Modality
Text-centric reasoning (thinking with text) excels at logic but struggles with spatial problems. Image-centric reasoning (thinking with images) handles space but lacks temporal dynamics. Video reasoning bridges both: generate videos showing problem-solving steps, with text embedded in frames. This unifies multimodal understanding within a single continuous temporal process.
The key insight is that video naturally represents processes—drawing, moving, transforming—that neither text nor static images capture well.
Core Concept
Video as Unified Reasoning Substrate:
- Dynamic Representation: Video captures continuous transformations over time
- Text Embedding: Embed reasoning text within video frames (e.g., solving equations step-by-step)
- Spatial Reasoning: Show spatial manipulations explicitly (moving, rotating, drawing)
- Temporal Reasoning: Natural representation of sequences and processes
- Single Model: One video generation model handles diverse reasoning tasks
Architecture Overview
- Video Generation Model: Pre-trained (Sora-2 or similar) for high-fidelity generation
- Task-Specific Prompting: Frame problems as video generation tasks
- Text-in-Video Encoding: Embed reasoning steps as video content
- Evaluation: Both spatial (puzzle accuracy) and mathematical (answer correctness)
Implementation Steps
1. Formulate Problem as Video Generation Task
Convert different problem types into video generation prompts.
class VideoReasoningPromptFormatting:
@staticmethod
def spatial_puzzle_to_video(puzzle_description):
"""Convert spatial puzzle to video generation prompt"""
prompt = f"""Generate a video showing the solution to this puzzle:
{puzzle_description}
The video should:
1. Start with the puzzle state
2. Show step-by-step manipulations
3. End with the solution
4. Include visual indicators (arrows, highlights) showing the transformations
"""
return prompt
@staticmethod
def math_problem_to_video(problem_text):
"""Convert math problem to video generation prompt"""
prompt = f"""Generate a video showing the step-by-step solution to this math problem:
{problem_text}
The video should:
1. Display the problem at the beginning
2. Show each reasoning step on screen
3. Include handwritten/typeset equations
4. Highlight key transitions between steps
5. Display the final answer prominently
"""
return prompt
@staticmethod
def eyeballing_puzzle_to_video(puzzle_image):
"""Convert visual puzzle to video generation prompt"""
prompt = f"""Generate a video showing how to solve this eyeballing puzzle:
[Image shown]
The video should:
1. Show the original puzzle
2. Demonstrate measurements or comparisons
3. Show the discovered relationship
4. End with the answer
"""
return prompt
2. Generate Videos and Extract Answers
Use video generation model as reasoning engine.
class VideoReasoningEngine:
def __init__(self, video_model, answer_extractor):
self.video_model = video_model # Sora-2 or similar
self.answer_extractor = answer_extractor
def solve_spatial_problem(self, puzzle_description):
# Convert problem to video prompt
prompt = VideoReasoningPromptFormatting.spatial_puzzle_to_video(puzzle_description)
# Generate video showing solution
video = self.video_model.generate(
prompt=prompt,
resolution="1280x720",
num_frames=24, # ~1 second at 24fps
duration=1.0
)
# Extract answer from final frame
final_frame = video.frames[-1]
answer = self.answer_extractor.extract_from_image(final_frame)
return answer, video
def solve_math_problem(self, problem_text):
# Convert problem to video prompt
prompt = VideoReasoningPromptFormatting.math_problem_to_video(problem_text)
# Generate video showing step-by-step solution
video = self.video_model.generate(
prompt=prompt,
resolution="1280x720",
num_frames=120, # ~5 seconds for more complex problems
duration=5.0
)
# Extract reasoning and answer from video frames
# Parse text that appears in frames
reasoning_steps = self.extract_reasoning_from_frames(video.frames)
final_answer = reasoning_steps[-1]
return final_answer, video, reasoning_steps
def extract_reasoning_from_frames(self, frames):
# Use OCR to extract text from frames
steps = []
for frame in frames:
text = pytesseract.image_to_string(frame)
if text.strip():
steps.append(text)
return steps
3. Implement Answer Extraction
Parse answers from generated video frames.
class VideoAnswerExtractor:
def __init__(self):
self.ocr_engine = pytesseract # Or cloud-based OCR
self.object_detector = ObjectDetector() # For spatial answers
def extract_from_image(self, frame):
# Try multiple extraction strategies
# Strategy 1: OCR
text = self.ocr_engine.image_to_string(frame)
if text and any(c.isdigit() for c in text):
return text
# Strategy 2: Object detection for spatial problems
objects = self.object_detector.detect(frame)
if objects:
# Extract spatial relationship
return self.describe_spatial_relationship(objects)
# Strategy 3: Color/shape analysis
return self.analyze_visual_properties(frame)
def extract_answer_from_video(self, video, problem_type):
"""Extract answer from entire video sequence"""
if problem_type == 'math':
# Parse all text frames for equations
frames_with_text = []
for frame in video.frames:
text = self.ocr_engine.image_to_string(frame)
if text:
frames_with_text.append(text)
final_line = frames_with_text[-1] if frames_with_text else ""
# Extract numeric answer
return self.extract_number(final_line)
elif problem_type == 'spatial':
# Use final frame(s) showing solution
final_frame = video.frames[-1]
return self.extract_from_image(final_frame)
def extract_number(self, text):
# Extract numeric answer from text
import re
numbers = re.findall(r'-?\d+\.?\d*', text)
return numbers[-1] if numbers else None
4. Benchmark on VideoThinkBench
Evaluate reasoning across different problem types.
class VideoReasoningBenchmark:
def __init__(self, engine, answer_extractor):
self.engine = engine
self.extractor = answer_extractor
def evaluate_spatial_tasks(self, spatial_problems):
"""Evaluate spatial reasoning (puzzles, mazes, etc.)"""
accuracies = []
for problem in spatial_problems:
predicted_answer, video = self.engine.solve_spatial_problem(problem['description'])
correct = predicted_answer == problem['ground_truth']
accuracies.append(correct)
return np.mean(accuracies)
def evaluate_math_tasks(self, math_problems):
"""Evaluate mathematical reasoning"""
accuracies = []
for problem in math_problems:
predicted_answer, video, steps = self.engine.solve_math_problem(
problem['problem_text']
)
correct = self.check_answer_equivalence(
predicted_answer, problem['ground_truth']
)
accuracies.append(correct)
return np.mean(accuracies)
def check_answer_equivalence(self, predicted, ground_truth):
# Handle different answer formats
try:
pred_val = float(predicted)
truth_val = float(ground_truth)
return abs(pred_val - truth_val) < 0.01
except:
return predicted.strip() == ground_truth.strip()
def run_full_benchmark(self, benchmark_dataset):
"""Run complete evaluation"""
results = {
'spatial': self.evaluate_spatial_tasks(benchmark_dataset['spatial']),
'math': self.evaluate_math_tasks(benchmark_dataset['math']),
'vision_centric': self.evaluate_spatial_tasks(benchmark_dataset['vision_centric']),
'text_centric': self.evaluate_math_tasks(benchmark_dataset['text_centric'])
}
return results
Practical Guidance
When to Use:
- Reasoning tasks combining spatial and textual elements
- Problem-solving that benefits from visual/temporal explanation
- Applications where reasoning process visibility is valuable
- Educational contexts (video explanations)
Hyperparameters:
- num_frames: 24-120 (more frames for complex reasoning)
- duration: 1-5 seconds (longer for detailed steps)
- resolution: 1280x720 (balance quality and inference cost)
When NOT to Use:
- Real-time reasoning (video generation is slow)
- Closed-book contexts (answer visible in generated video violates some setups)
- High-frequency reasoning (too expensive per problem)
Pitfalls:
- Answer leakage: Generated video may show answer too early; control frame ordering
- OCR errors: Extracting text from video frames is fragile; use robust OCR
- Incoherent videos: Video models can generate unrealistic visualizations; validate plausibility
Integration Strategy: Use as auxiliary reasoning modality alongside text. For hybrid tasks, generate both text reasoning and video reasoning, then ensemble answers.