# Visplay Self Evolving Vlms > Enable VLMs to self-improve from unlabeled images via dual-role framework—questioner generates challenging visual questions while reasoner answers them, trained jointly with GRPO using difficulty and diversity rewards. - Skill: `adu2021/visplay-self-evolving-vlms` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/visplay-self-evolving-vlms` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/visplay-self-evolving-vlms/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/visplay-self-evolving-vlms --- # Enable Vision-Language Models to Self-Improve from Unlabeled Images Vision-language models typically require supervised fine-tuning on annotated data. VisPlay breaks this dependency via a **dual-role self-evolution framework**: the model splits into an Image-Conditioned Questioner (generates visual questions) and a Multimodal Reasoner (answers them), trained jointly with GRPO. The system autonomously creates an automatic curriculum from unlabeled images, progressively increasing difficulty. No annotations needed—the questioner and reasoner co-evolve, with the questioner learning to generate harder questions as the reasoner improves, creating a self-reinforcing loop of increasing capability. ## Core Concept Supervised learning for VLMs requires annotated data (image-question-answer triples), which is expensive. VisPlay observes that annotators follow a natural curriculum: they ask easier questions first, then harder ones as models improve. VisPlay automates this: 1. **Questioner Role**: Given an image, generate a challenging visual question about it 2. **Reasoner Role**: Answer the question using both image and question context Both roles start as the same base model. During training: - Questioner learns to generate questions of optimal difficulty (50% confidence for reasoner) - Reasoner learns to answer increasingly difficult questions They're trained jointly with GRPO, using composite rewards (uncertainty reward, diversity, format constraints). This creates an automatic curriculum without human annotation. ## Architecture Overview - **Dual-Role Architecture**: Single base model splits into Questioner and Reasoner for alternating training - **Questioner Rewards**: Uncertainty (target 50% reasoner confidence), diversity (penalize redundancy), format (enforce structure) - **Reasoner Rewards**: Binary (correct/incorrect) based on pseudo-labels from majority voting on reasoner outputs - **GRPO Training**: Group Relative Policy Optimization for stable reward signal without separate value network - **Curriculum Learning**: Automatic difficulty progression as reasoner improves, questioner escalates ## Implementation Steps **Step 1: Define Questioner and Reasoner Roles.** Split model into two heads. ```python import torch import torch.nn as nn class DualRoleVLM(nn.Module): """ Vision-language model with questioner and reasoner roles. Both use shared backbone; different heads for generation type. """ def __init__(self, base_model_name='clip-vit-base', hidden_dim=768): super().__init__() # Shared vision-language backbone self.backbone = load_vision_language_model(base_model_name) # Role-specific heads self.questioner_head = nn.Linear(hidden_dim, hidden_dim) self.reasoner_head = nn.Linear(hidden_dim, hidden_dim) # Shared decoder for generation self.decoder = nn.TransformerDecoder( decoder_layer=nn.TransformerDecoderLayer(hidden_dim, nhead=8), num_layers=6 ) self.output_projection = nn.Linear(hidden_dim, 32000) # Vocab size def forward_questioner(self, images): """ Generate question given image. Returns: question tokens and generation logits """ # Encode image img_features = self.backbone.encode_image(images) # (batch, hidden_dim) # Route through questioner head questioner_features = self.questioner_head(img_features) # Generate question via autoregressive decoding questions, logits = self._generate_sequence( questioner_features, max_length=20, prompt="Question: " ) return questions, logits def forward_reasoner(self, images, questions): """ Answer question given image and question. Returns: answer tokens and generation logits """ # Encode image and question img_features = self.backbone.encode_image(images) question_tokens = self.backbone.encode_text(questions) # Route through reasoner head reasoner_features = self.reasoner_head(img_features) # Concatenate with question combined_features = torch.cat([reasoner_features, question_tokens], dim=1) # Generate answer answers, logits = self._generate_sequence( combined_features, max_length=30, prompt="Answer: " ) return answers, logits def _generate_sequence(self, features, max_length, prompt=""): """ Autoregressive generation helper. """ batch_size = features.shape[0] tokens = [] logits_list = [] # Decode autoregressively for _ in range(max_length): # Decoder forward pass output = self.decoder(features.unsqueeze(0)) # (1, batch, hidden) output_logits = self.output_projection(output[-1]) # (batch, vocab) # Sample token (greedy or sampling) next_token = output_logits.argmax(dim=-1) tokens.append(next_token) logits_list.append(output_logits) # Check for end-of-sequence if (next_token == self.get_eos_token()).all(): break return torch.stack(tokens, dim=1), torch.stack(logits_list, dim=1) ``` **Step 2: Define Reward Functions for Questioner and Reasoner.** ```python class DualRoleRewardFunction: def __init__(self): self.difficulty_history = [] def reward_questioner(self, question, reasoner_confidence, previous_confidence): """ Reward questioner for generating questions of optimal difficulty. Optimal = 50% reasoner confidence (neither too easy nor too hard) """ # Uncertainty reward: target 50% confidence uncertainty_reward = 1.0 - 2 * abs(0.5 - reasoner_confidence) # Difficulty progression reward # Bonus if difficulty increased (shows curriculum learning) difficulty_increase = reasoner_confidence - previous_confidence progression_reward = 0.5 if difficulty_increase < 0 else 0.0 # Negative = harder # Format/structure reward (enforced via constraints) format_reward = 0.5 # Assume format is valid; penalize invalid separately total_reward = 0.6 * uncertainty_reward + 0.3 * progression_reward + 0.1 * format_reward return total_reward def reward_reasoner(self, answer, ground_truth, confidence): """ Reward reasoner for correct answers. Use majority-voted pseudo-labels as ground truth. """ is_correct = self._evaluate_answer(answer, ground_truth) if is_correct: # Bonus for high confidence when correct return 1.0 + 0.2 * confidence else: # Penalty, worse with high confidence (overconfidence) return -1.0 - 0.2 * confidence def _evaluate_answer(self, answer, ground_truth): """ Simple evaluation: exact match or semantic similarity. For unlabeled data, use majority voting on k reasoner outputs. """ # Placeholder: use BERT embedding similarity return answer.lower().strip() == ground_truth.lower().strip() def get_difficulty_score(self, confidences): """Track difficulty progression over time.""" avg_confidence = sum(confidences) / len(confidences) self.difficulty_history.append(avg_confidence) return avg_confidence ``` **Step 3: GRPO Training Loop.** Train questioner and reasoner jointly. ```python def train_dual_role_with_grpo( model, image_dataloader, num_iterations=10000, lr=1e-5 ): """ Train vision-language model with dual-role GRPO. """ optimizer = torch.optim.AdamW(model.parameters(), lr=lr) reward_fn = DualRoleRewardFunction() for iteration in range(num_iterations): for images in image_dataloader: batch_size = images.shape[0] # ===== Questioner Phase ===== # Generate questions questions, questioner_logits = model.forward_questioner(images) # ===== Reasoner Phase ===== # Answer questions to get rewards answers, reasoner_logits = model.forward_reasoner(images, questions) # Compute reasoner confidence (softmax on output logits) reasoner_confidence = torch.softmax(reasoner_logits, dim=-1).max(dim=-1).values.mean() # ===== Pseudo-Labeling ===== # For unlabeled data, use majority voting on multiple reasoner runs pseudo_labels = self._majority_vote_labels( model, images, questions, num_runs=5 ) # ===== Reward Computation ===== questioner_rewards = [] reasoner_rewards = [] for i in range(batch_size): # Questioner reward (difficulty calibration) q_reward = reward_fn.reward_questioner( questions[i], reasoner_confidence.item(), prev_confidence=0.5 # Running average ) questioner_rewards.append(q_reward) # Reasoner reward (answer correctness) r_reward = reward_fn.reward_reasoner( answers[i], pseudo_labels[i], confidence=reasoner_confidence.item() ) reasoner_rewards.append(r_reward) questioner_rewards = torch.tensor(questioner_rewards) reasoner_rewards = torch.tensor(reasoner_rewards) # ===== GRPO: Group Relative Policy Optimization ===== # Compute advantages as relative performance within group q_baseline = questioner_rewards.mean() q_advantages = (questioner_rewards - q_baseline) / (questioner_rewards.std() + 1e-8) r_baseline = reasoner_rewards.mean() r_advantages = (reasoner_rewards - r_baseline) / (reasoner_rewards.std() + 1e-8) # Policy gradient losses q_log_probs = torch.log_softmax(questioner_logits, dim=-1).mean(dim=1) q_loss = -(q_log_probs * q_advantages).mean() r_log_probs = torch.log_softmax(reasoner_logits, dim=-1).mean(dim=1) r_loss = -(r_log_probs * r_advantages).mean() total_loss = q_loss + r_loss # Backward pass optimizer.zero_grad() total_loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.step() if iteration % 100 == 0: print(f"Iter {iteration}: q_loss={q_loss:.4f}, r_loss={r_loss:.4f}, " f"confidence={reasoner_confidence:.3f}") def _majority_vote_labels(self, model, images, questions, num_runs=5): """Pseudo-label generation via majority voting.""" all_answers = [] for _ in range(num_runs): answers, _ = model.forward_reasoner(images, questions) all_answers.append(answers) # Majority vote labels = [] for i in range(images.shape[0]): answers_for_sample = [ans[i] for ans in all_answers] majority_answer = self._find_majority(answers_for_sample) labels.append(majority_answer) return labels ``` ## Practical Guidance **When to Use:** Self-improvement of VLMs when labeled data is scarce or expensive. Use for domains with many unlabeled images (web, internal datasets). **Hyperparameters:** - Uncertainty target: 50% confidence optimal; adjust to 40-60% range for harder curriculum - Group size for GRPO: batch size ≥ 32; larger batches stabilize advantage estimation - Confidence thresholds for reasoner: 0.25–0.75 for filtering moderately-confident examples; avoids learning from noise **Pitfalls:** - **Pseudo-label quality**: Majority voting with small k (< 5 runs) is noisy; use k ≥ 5, consider semantic clustering - **Question degeneracy**: Questioner may converge to trivial questions; use diversity penalty aggressively - **Confidence calibration**: Model confidence may not correlate with correctness; validate on labeled data periodically - **Convergence oscillation**: Questioner and reasoner can oscillate; use EMA for stability **When NOT to Use:** Tasks with abundant labeled data (supervised fine-tuning is simpler); domains where pseudo-labels are unreliable. **Integration**: Works with any VLM backbone (CLIP, LLaVA, GPT-4V-style models); no architectural changes needed. --- Reference: https://arxiv.org/abs/2511.15661