# Thinking With Video Multimodal Reasoning > Leverage video generation models as unified multimodal reasoning engines that bridge text and vision by embedding reasoning as dynamic visual processes, enabling models to tackle both spatial puzzles and mathematical problems within a single temporal framework. - Skill: `adu2021/thinking-with-video-multimodal-reasoning` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/thinking-with-video-multimodal-reasoning` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/thinking-with-video-multimodal-reasoning/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/thinking-with-video-multimodal-reasoning --- # Title: Use Video Generation as a Universal Reasoning Modality Text-centric reasoning (thinking with text) excels at logic but struggles with spatial problems. Image-centric reasoning (thinking with images) handles space but lacks temporal dynamics. Video reasoning bridges both: generate videos showing problem-solving steps, with text embedded in frames. This unifies multimodal understanding within a single continuous temporal process. The key insight is that video naturally represents processes—drawing, moving, transforming—that neither text nor static images capture well. ## Core Concept **Video as Unified Reasoning Substrate**: - **Dynamic Representation**: Video captures continuous transformations over time - **Text Embedding**: Embed reasoning text within video frames (e.g., solving equations step-by-step) - **Spatial Reasoning**: Show spatial manipulations explicitly (moving, rotating, drawing) - **Temporal Reasoning**: Natural representation of sequences and processes - **Single Model**: One video generation model handles diverse reasoning tasks ## Architecture Overview - **Video Generation Model**: Pre-trained (Sora-2 or similar) for high-fidelity generation - **Task-Specific Prompting**: Frame problems as video generation tasks - **Text-in-Video Encoding**: Embed reasoning steps as video content - **Evaluation**: Both spatial (puzzle accuracy) and mathematical (answer correctness) ## Implementation Steps **1. Formulate Problem as Video Generation Task** Convert different problem types into video generation prompts. ```python class VideoReasoningPromptFormatting: @staticmethod def spatial_puzzle_to_video(puzzle_description): """Convert spatial puzzle to video generation prompt""" prompt = f"""Generate a video showing the solution to this puzzle: {puzzle_description} The video should: 1. Start with the puzzle state 2. Show step-by-step manipulations 3. End with the solution 4. Include visual indicators (arrows, highlights) showing the transformations """ return prompt @staticmethod def math_problem_to_video(problem_text): """Convert math problem to video generation prompt""" prompt = f"""Generate a video showing the step-by-step solution to this math problem: {problem_text} The video should: 1. Display the problem at the beginning 2. Show each reasoning step on screen 3. Include handwritten/typeset equations 4. Highlight key transitions between steps 5. Display the final answer prominently """ return prompt @staticmethod def eyeballing_puzzle_to_video(puzzle_image): """Convert visual puzzle to video generation prompt""" prompt = f"""Generate a video showing how to solve this eyeballing puzzle: [Image shown] The video should: 1. Show the original puzzle 2. Demonstrate measurements or comparisons 3. Show the discovered relationship 4. End with the answer """ return prompt ``` **2. Generate Videos and Extract Answers** Use video generation model as reasoning engine. ```python class VideoReasoningEngine: def __init__(self, video_model, answer_extractor): self.video_model = video_model # Sora-2 or similar self.answer_extractor = answer_extractor def solve_spatial_problem(self, puzzle_description): # Convert problem to video prompt prompt = VideoReasoningPromptFormatting.spatial_puzzle_to_video(puzzle_description) # Generate video showing solution video = self.video_model.generate( prompt=prompt, resolution="1280x720", num_frames=24, # ~1 second at 24fps duration=1.0 ) # Extract answer from final frame final_frame = video.frames[-1] answer = self.answer_extractor.extract_from_image(final_frame) return answer, video def solve_math_problem(self, problem_text): # Convert problem to video prompt prompt = VideoReasoningPromptFormatting.math_problem_to_video(problem_text) # Generate video showing step-by-step solution video = self.video_model.generate( prompt=prompt, resolution="1280x720", num_frames=120, # ~5 seconds for more complex problems duration=5.0 ) # Extract reasoning and answer from video frames # Parse text that appears in frames reasoning_steps = self.extract_reasoning_from_frames(video.frames) final_answer = reasoning_steps[-1] return final_answer, video, reasoning_steps def extract_reasoning_from_frames(self, frames): # Use OCR to extract text from frames steps = [] for frame in frames: text = pytesseract.image_to_string(frame) if text.strip(): steps.append(text) return steps ``` **3. Implement Answer Extraction** Parse answers from generated video frames. ```python class VideoAnswerExtractor: def __init__(self): self.ocr_engine = pytesseract # Or cloud-based OCR self.object_detector = ObjectDetector() # For spatial answers def extract_from_image(self, frame): # Try multiple extraction strategies # Strategy 1: OCR text = self.ocr_engine.image_to_string(frame) if text and any(c.isdigit() for c in text): return text # Strategy 2: Object detection for spatial problems objects = self.object_detector.detect(frame) if objects: # Extract spatial relationship return self.describe_spatial_relationship(objects) # Strategy 3: Color/shape analysis return self.analyze_visual_properties(frame) def extract_answer_from_video(self, video, problem_type): """Extract answer from entire video sequence""" if problem_type == 'math': # Parse all text frames for equations frames_with_text = [] for frame in video.frames: text = self.ocr_engine.image_to_string(frame) if text: frames_with_text.append(text) final_line = frames_with_text[-1] if frames_with_text else "" # Extract numeric answer return self.extract_number(final_line) elif problem_type == 'spatial': # Use final frame(s) showing solution final_frame = video.frames[-1] return self.extract_from_image(final_frame) def extract_number(self, text): # Extract numeric answer from text import re numbers = re.findall(r'-?\d+\.?\d*', text) return numbers[-1] if numbers else None ``` **4. Benchmark on VideoThinkBench** Evaluate reasoning across different problem types. ```python class VideoReasoningBenchmark: def __init__(self, engine, answer_extractor): self.engine = engine self.extractor = answer_extractor def evaluate_spatial_tasks(self, spatial_problems): """Evaluate spatial reasoning (puzzles, mazes, etc.)""" accuracies = [] for problem in spatial_problems: predicted_answer, video = self.engine.solve_spatial_problem(problem['description']) correct = predicted_answer == problem['ground_truth'] accuracies.append(correct) return np.mean(accuracies) def evaluate_math_tasks(self, math_problems): """Evaluate mathematical reasoning""" accuracies = [] for problem in math_problems: predicted_answer, video, steps = self.engine.solve_math_problem( problem['problem_text'] ) correct = self.check_answer_equivalence( predicted_answer, problem['ground_truth'] ) accuracies.append(correct) return np.mean(accuracies) def check_answer_equivalence(self, predicted, ground_truth): # Handle different answer formats try: pred_val = float(predicted) truth_val = float(ground_truth) return abs(pred_val - truth_val) < 0.01 except: return predicted.strip() == ground_truth.strip() def run_full_benchmark(self, benchmark_dataset): """Run complete evaluation""" results = { 'spatial': self.evaluate_spatial_tasks(benchmark_dataset['spatial']), 'math': self.evaluate_math_tasks(benchmark_dataset['math']), 'vision_centric': self.evaluate_spatial_tasks(benchmark_dataset['vision_centric']), 'text_centric': self.evaluate_math_tasks(benchmark_dataset['text_centric']) } return results ``` ## Practical Guidance **When to Use**: - Reasoning tasks combining spatial and textual elements - Problem-solving that benefits from visual/temporal explanation - Applications where reasoning process visibility is valuable - Educational contexts (video explanations) **Hyperparameters**: - num_frames: 24-120 (more frames for complex reasoning) - duration: 1-5 seconds (longer for detailed steps) - resolution: 1280x720 (balance quality and inference cost) **When NOT to Use**: - Real-time reasoning (video generation is slow) - Closed-book contexts (answer visible in generated video violates some setups) - High-frequency reasoning (too expensive per problem) **Pitfalls**: - **Answer leakage**: Generated video may show answer too early; control frame ordering - **OCR errors**: Extracting text from video frames is fragile; use robust OCR - **Incoherent videos**: Video models can generate unrealistic visualizations; validate plausibility **Integration Strategy**: Use as auxiliary reasoning modality alongside text. For hybrid tasks, generate both text reasoning and video reasoning, then ensemble answers. ## Reference arXiv: https://arxiv.org/abs/2511.04570