# V Thinker Interactive Visual Reasoning > Enable vision-centric interactive reasoning by synthesizing diverse reasoning datasets through co-evolutionary loops, then training models with progressive curriculum that starts with perception and advances to tool-based problem solving. - Skill: `adu2021/v-thinker-interactive-visual-reasoning` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/v-thinker-interactive-visual-reasoning` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/v-thinker-interactive-visual-reasoning/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/v-thinker-interactive-visual-reasoning --- # Title: Train Models to Think Visually Through Interactive Tool Use Standard vision-language models can recognize objects but struggle with interactive visual reasoning—using tools like pointing, drawing, zooming to solve problems. V-Thinker bootstraps this capability through three key mechanisms: (1) data evolution flywheel that synthesizes high-quality reasoning datasets across dimensions of variety/quality/difficulty, (2) progressive training curriculum from perception to interaction, and (3) reward-shaped RL that incentivizes effective tool use. The approach transforms VLMs from passive observers into active problem-solvers. ## Core Concept **Data Evolution and Progressive Training for Visual Interaction**: - **Data Flywheel**: Knowledge concepts and tool use co-evolve, expanding diversity - **Quality Calibration**: Checker-repairer loop maintains correctness - **Difficulty Progression**: Systematically increase problem complexity - **Perception First**: SFT on basic spatial understanding before tool use - **Interactive RL**: GRPO optimizes for effective reasoning and tool application ## Architecture Overview - **Data Synthesis Engine**: Automatically generates varied, high-quality reasoning tasks - **V-Perception-40K Dataset**: Foundational perception with point-level supervision - **V-Interaction-400K Dataset**: Evolved interactive reasoning examples - **Progressive Curriculum**: Two-stage training (perception → interaction) - **VTBench**: 1,500 expert-reviewed problems for evaluation ## Implementation Steps **1. Implement Data Evolution Flywheel** Create system that automatically synthesizes and improves training data. ```python class DataEvolutionFlywheel: def __init__(self, base_concepts, base_tools): self.concepts = set(base_concepts) self.tools = set(base_tools) def expand_concepts(self): """Automatically generate new knowledge concepts""" # Existing concepts: ["geometry", "color", "counting"] # Generate combinations and variations new_concepts = [] for concept_pair in combinations(self.concepts, 2): hybrid = f"{concept_pair[0]}_and_{concept_pair[1]}" new_concepts.append(hybrid) # Add specialized variants for concept in list(self.concepts): for modifier in ["advanced", "inverse", "composite"]: new_concepts.append(f"{modifier}_{concept}") self.concepts.update(new_concepts) return new_concepts def expand_tools(self): """Discover new visual tools needed for new concepts""" # If we need to reason about "line_intersection", add "drawing" tool new_tools = [] for concept in self.concepts: if "intersection" in concept and "drawing" not in self.tools: new_tools.append("drawing") if "distance" in concept and "measurement" not in self.tools: new_tools.append("measurement") if "rotation" in concept and "rotation_ui" not in self.tools: new_tools.append("rotation_ui") self.tools.update(new_tools) return new_tools def generate_problems(self, num_problems=1000): """Generate reasoning problems using expanded concepts and tools""" problems = [] for _ in range(num_problems): # Sample concept and tool concept = random.choice(list(self.concepts)) required_tools = self.tools_for_concept(concept) # Generate problem problem = { 'concept': concept, 'required_tools': required_tools, 'image': self.generate_problem_image(concept), 'question': self.generate_question(concept), 'answer': None # To be filled in } problems.append(problem) return problems def tools_for_concept(self, concept): """Map concepts to required tools""" mapping = { 'geometry': ['pointing', 'drawing'], 'color': ['highlighting'], 'counting': ['pointing'], 'distance': ['measurement'], 'rotation': ['rotation_ui'] } return mapping.get(concept, ['pointing']) ``` **2. Implement Quality Calibration** Use checker-repairer loop to ensure data correctness. ```python class QualityCalibration: def __init__(self, model): self.model = model # VLM for checking def check_problem(self, problem): """Verify problem is well-formed and correctly answered""" checks = { 'image_valid': self.check_image(problem['image']), 'question_clear': self.check_question(problem['question']), 'answer_correct': self.check_answer(problem), 'tools_necessary': self.check_tool_necessity(problem) } all_valid = all(checks.values()) return all_valid, checks def repair_problem(self, problem, failed_checks): """Repair problems that failed checks""" if not failed_checks['answer_correct']: # Regenerate answer problem['answer'] = self.regenerate_answer(problem) if not failed_checks['question_clear']: # Rephrase question problem['question'] = self.rephrase_question(problem) if not failed_checks['tools_necessary']: # Adjust problem to make tool use necessary problem['complexity'] += 1 return problem def calibration_loop(self, problems, max_repairs=2): """Run check-repair cycle""" calibrated = [] for problem in problems: repairs = 0 while repairs < max_repairs: valid, checks = self.check_problem(problem) if valid: break problem = self.repair_problem(problem, checks) repairs += 1 if valid: calibrated.append(problem) return calibrated ``` **3. Implement Progressive Training Curriculum** Two-stage training: perception first, then interaction. ```python class ProgressiveTrainingCurriculum: def __init__(self, model): self.model = model def stage1_perception(self, dataset_perception, num_epochs=3): """SFT on perception with point-level supervision""" # dataset_perception: V-Perception-40K # Contains basic spatial reasoning with point annotations optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-4) for epoch in range(num_epochs): for batch in dataset_perception: images, questions, points, answers = batch # Loss includes point localization output = self.model(images, questions) # SFT loss sft_loss = cross_entropy_loss(output.logits, answers) # Point supervision loss point_loss = point_localization_loss(output.points, points) loss = sft_loss + 0.5 * point_loss optimizer.zero_grad() loss.backward() optimizer.step() def stage2_interaction(self, dataset_interaction, num_steps=10000): """RL with GRPO on interactive reasoning""" # dataset_interaction: V-Interaction-400K # Requires tool use: drawing, measurement, zooming, etc. for step in range(num_steps): batch = dataset_interaction.sample_batch() images, questions, answers = batch # Generate reasoning trajectories trajectories = [] for i, (image, question) in enumerate(zip(images, questions)): trajectory = self.model.generate_interactive_solution(image, question) trajectories.append(trajectory) # Compute rewards rewards = [] for trajectory, answer in zip(trajectories, answers): # Accuracy reward accuracy = trajectory.final_answer == answer # Format reward: proper tool use format_correct = self.check_tool_format(trajectory) # Efficiency reward: fewer steps is better efficiency = 1.0 / (1.0 + len(trajectory.steps)) reward = 0.6 * accuracy + 0.3 * format_correct + 0.1 * efficiency rewards.append(reward) # GRPO optimization loss = self.compute_grpo_loss(trajectories, torch.tensor(rewards)) self.optimizer.zero_grad() loss.backward() self.optimizer.step() ``` **4. Create VTBench for Evaluation** Build expert-reviewed benchmark for vision-centric reasoning. ```python class VTBench: def __init__(self, num_problems=1500): self.problems = [] self.expertiser_reviews = {} def create_problems_from_domains(self): """Sample problems across domains""" domains = ['geometry', 'counting', 'spatial_relations', 'color_reasoning', 'measurement'] for domain in domains: domain_problems = self.generate_domain_problems(domain, num=300) for problem in domain_problems: # Human expert validation expert_review = self.collect_expert_review(problem) if expert_review['is_valid']: problem['difficulty'] = expert_review['difficulty'] problem['requires_tools'] = expert_review['tools'] self.problems.append(problem) def collect_expert_review(self, problem): """Collect expert annotations""" return { 'is_valid': True, 'difficulty': random.choice([1, 2, 3]), # 1-3 scale 'tools': self.infer_required_tools(problem) } def evaluate_model(self, model): """Benchmark model on VTBench""" accuracies = [] for problem in self.problems: image = problem['image'] question = problem['question'] expected_answer = problem['answer'] # Get model solution predicted_answer = model.solve(image, question) # Check correctness correct = predicted_answer == expected_answer accuracies.append(correct) return np.mean(accuracies) ``` ## Practical Guidance **When to Use**: - Fine-tuning VLMs for interactive visual reasoning - Tasks requiring tool use (drawing, measuring, manipulating) - Agents that need to explore visual scenes **Hyperparameters**: - num_concepts_initial: 5-10 (start with core concepts) - expansion_factor: 2-3 (multiply concepts/tools per iteration) - perception_epochs: 2-3 (brief warm-up) - grpo_steps: 10K-50K (scale with dataset size) **When NOT to Use**: - Passive image understanding (no interactive tools needed) - Very simple tasks where SFT alone suffices - Low-data regimes (data evolution requires significant volume) **Pitfalls**: - **Tool use degeneracy**: Model learns to invoke tools randomly; require tool necessity - **Dataset bootstrap failure**: If base concepts are poorly chosen, evolution stagnates - **RL instability**: Large reward swings can cause divergence; use reward clipping **Integration Strategy**: Apply as fine-tuning layer on top of pre-trained VLMs. Use both stages—perception alone gives weak results. ## Reference arXiv: https://arxiv.org/abs/2511.04460