adu2021 Enable Vision-Language Models to Self-Improve from Unlabeled Images
Vision-language models typically require supervised fine-tuning on annotated data. VisPlay breaks this dependency via a dual-role self-evolution framework: the model splits into an Image-Conditioned Questioner (generates visual questions) and a Multimodal Reasoner (answers them), trained jointly with GRPO. The system autonomously creates an automatic curriculum from unlabeled images, progressively increasing difficulty.
No annotations needed—the questioner and reasoner co-evolve, with the questioner learning to generate harder questions as the reasoner improves, creating a self-reinforcing loop of increasing capability.
Core Concept
Supervised learning for VLMs requires annotated data (image-question-answer triples), which is expensive. VisPlay observes that annotators follow a natural curriculum: they ask easier questions first, then harder ones as models improve. VisPlay automates this:
- Questioner Role: Given an image, generate a challenging visual question about it
- Reasoner Role: Answer the question using both image and question context
Both roles start as the same base model. During training:
- Questioner learns to generate questions of optimal difficulty (50% confidence for reasoner)
- Reasoner learns to answer increasingly difficult questions
They're trained jointly with GRPO, using composite rewards (uncertainty reward, diversity, format constraints). This creates an automatic curriculum without human annotation.
Architecture Overview
- Dual-Role Architecture: Single base model splits into Questioner and Reasoner for alternating training
- Questioner Rewards: Uncertainty (target 50% reasoner confidence), diversity (penalize redundancy), format (enforce structure)
- Reasoner Rewards: Binary (correct/incorrect) based on pseudo-labels from majority voting on reasoner outputs
- GRPO Training: Group Relative Policy Optimization for stable reward signal without separate value network
- Curriculum Learning: Automatic difficulty progression as reasoner improves, questioner escalates
Implementation Steps
Step 1: Define Questioner and Reasoner Roles. Split model into two heads.
import torch
import torch.nn as nn
class DualRoleVLM(nn.Module):
"""
Vision-language model with questioner and reasoner roles.
Both use shared backbone; different heads for generation type.
"""
def __init__(self, base_model_name='clip-vit-base', hidden_dim=768):
super().__init__()
# Shared vision-language backbone
self.backbone = load_vision_language_model(base_model_name)
# Role-specific heads
self.questioner_head = nn.Linear(hidden_dim, hidden_dim)
self.reasoner_head = nn.Linear(hidden_dim, hidden_dim)
# Shared decoder for generation
self.decoder = nn.TransformerDecoder(
decoder_layer=nn.TransformerDecoderLayer(hidden_dim, nhead=8),
num_layers=6
)
self.output_projection = nn.Linear(hidden_dim, 32000) # Vocab size
def forward_questioner(self, images):
"""
Generate question given image.
Returns: question tokens and generation logits
"""
# Encode image
img_features = self.backbone.encode_image(images) # (batch, hidden_dim)
# Route through questioner head
questioner_features = self.questioner_head(img_features)
# Generate question via autoregressive decoding
questions, logits = self._generate_sequence(
questioner_features,
max_length=20,
prompt="Question: "
)
return questions, logits
def forward_reasoner(self, images, questions):
"""
Answer question given image and question.
Returns: answer tokens and generation logits
"""
# Encode image and question
img_features = self.backbone.encode_image(images)
question_tokens = self.backbone.encode_text(questions)
# Route through reasoner head
reasoner_features = self.reasoner_head(img_features)
# Concatenate with question
combined_features = torch.cat([reasoner_features, question_tokens], dim=1)
# Generate answer
answers, logits = self._generate_sequence(
combined_features,
max_length=30,
prompt="Answer: "
)
return answers, logits
def _generate_sequence(self, features, max_length, prompt=""):
"""
Autoregressive generation helper.
"""
batch_size = features.shape[0]
tokens = []
logits_list = []
# Decode autoregressively
for _ in range(max_length):
# Decoder forward pass
output = self.decoder(features.unsqueeze(0)) # (1, batch, hidden)
output_logits = self.output_projection(output[-1]) # (batch, vocab)
# Sample token (greedy or sampling)
next_token = output_logits.argmax(dim=-1)
tokens.append(next_token)
logits_list.append(output_logits)
# Check for end-of-sequence
if (next_token == self.get_eos_token()).all():
break
return torch.stack(tokens, dim=1), torch.stack(logits_list, dim=1)
Step 2: Define Reward Functions for Questioner and Reasoner.
class DualRoleRewardFunction:
def __init__(self):
self.difficulty_history = []
def reward_questioner(self, question, reasoner_confidence, previous_confidence):
"""
Reward questioner for generating questions of optimal difficulty.
Optimal = 50% reasoner confidence (neither too easy nor too hard)
"""
# Uncertainty reward: target 50% confidence
uncertainty_reward = 1.0 - 2 * abs(0.5 - reasoner_confidence)
# Difficulty progression reward
# Bonus if difficulty increased (shows curriculum learning)
difficulty_increase = reasoner_confidence - previous_confidence
progression_reward = 0.5 if difficulty_increase < 0 else 0.0 # Negative = harder
# Format/structure reward (enforced via constraints)
format_reward = 0.5 # Assume format is valid; penalize invalid separately
total_reward = 0.6 * uncertainty_reward + 0.3 * progression_reward + 0.1 * format_reward
return total_reward
def reward_reasoner(self, answer, ground_truth, confidence):
"""
Reward reasoner for correct answers.
Use majority-voted pseudo-labels as ground truth.
"""
is_correct = self._evaluate_answer(answer, ground_truth)
if is_correct:
# Bonus for high confidence when correct
return 1.0 + 0.2 * confidence
else:
# Penalty, worse with high confidence (overconfidence)
return -1.0 - 0.2 * confidence
def _evaluate_answer(self, answer, ground_truth):
"""
Simple evaluation: exact match or semantic similarity.
For unlabeled data, use majority voting on k reasoner outputs.
"""
# Placeholder: use BERT embedding similarity
return answer.lower().strip() == ground_truth.lower().strip()
def get_difficulty_score(self, confidences):
"""Track difficulty progression over time."""
avg_confidence = sum(confidences) / len(confidences)
self.difficulty_history.append(avg_confidence)
return avg_confidence
Step 3: GRPO Training Loop. Train questioner and reasoner jointly.
def train_dual_role_with_grpo(
model, image_dataloader, num_iterations=10000, lr=1e-5
):
"""
Train vision-language model with dual-role GRPO.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
reward_fn = DualRoleRewardFunction()
for iteration in range(num_iterations):
for images in image_dataloader:
batch_size = images.shape[0]
# ===== Questioner Phase =====
# Generate questions
questions, questioner_logits = model.forward_questioner(images)
# ===== Reasoner Phase =====
# Answer questions to get rewards
answers, reasoner_logits = model.forward_reasoner(images, questions)
# Compute reasoner confidence (softmax on output logits)
reasoner_confidence = torch.softmax(reasoner_logits, dim=-1).max(dim=-1).values.mean()
# ===== Pseudo-Labeling =====
# For unlabeled data, use majority voting on multiple reasoner runs
pseudo_labels = self._majority_vote_labels(
model, images, questions, num_runs=5
)
# ===== Reward Computation =====
questioner_rewards = []
reasoner_rewards = []
for i in range(batch_size):
# Questioner reward (difficulty calibration)
q_reward = reward_fn.reward_questioner(
questions[i],
reasoner_confidence.item(),
prev_confidence=0.5 # Running average
)
questioner_rewards.append(q_reward)
# Reasoner reward (answer correctness)
r_reward = reward_fn.reward_reasoner(
answers[i],
pseudo_labels[i],
confidence=reasoner_confidence.item()
)
reasoner_rewards.append(r_reward)
questioner_rewards = torch.tensor(questioner_rewards)
reasoner_rewards = torch.tensor(reasoner_rewards)
# ===== GRPO: Group Relative Policy Optimization =====
# Compute advantages as relative performance within group
q_baseline = questioner_rewards.mean()
q_advantages = (questioner_rewards - q_baseline) / (questioner_rewards.std() + 1e-8)
r_baseline = reasoner_rewards.mean()
r_advantages = (reasoner_rewards - r_baseline) / (reasoner_rewards.std() + 1e-8)
# Policy gradient losses
q_log_probs = torch.log_softmax(questioner_logits, dim=-1).mean(dim=1)
q_loss = -(q_log_probs * q_advantages).mean()
r_log_probs = torch.log_softmax(reasoner_logits, dim=-1).mean(dim=1)
r_loss = -(r_log_probs * r_advantages).mean()
total_loss = q_loss + r_loss
# Backward pass
optimizer.zero_grad()
total_loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
if iteration % 100 == 0:
print(f"Iter {iteration}: q_loss={q_loss:.4f}, r_loss={r_loss:.4f}, "
f"confidence={reasoner_confidence:.3f}")
def _majority_vote_labels(self, model, images, questions, num_runs=5):
"""Pseudo-label generation via majority voting."""
all_answers = []
for _ in range(num_runs):
answers, _ = model.forward_reasoner(images, questions)
all_answers.append(answers)
# Majority vote
labels = []
for i in range(images.shape[0]):
answers_for_sample = [ans[i] for ans in all_answers]
majority_answer = self._find_majority(answers_for_sample)
labels.append(majority_answer)
return labels
Practical Guidance
When to Use: Self-improvement of VLMs when labeled data is scarce or expensive. Use for domains with many unlabeled images (web, internal datasets).
Hyperparameters:
- Uncertainty target: 50% confidence optimal; adjust to 40-60% range for harder curriculum
- Group size for GRPO: batch size ≥ 32; larger batches stabilize advantage estimation
- Confidence thresholds for reasoner: 0.25–0.75 for filtering moderately-confident examples; avoids learning from noise
Pitfalls:
- Pseudo-label quality: Majority voting with small k (< 5 runs) is noisy; use k ≥ 5, consider semantic clustering
- Question degeneracy: Questioner may converge to trivial questions; use diversity penalty aggressively
- Confidence calibration: Model confidence may not correlate with correctness; validate on labeled data periodically
- Convergence oscillation: Questioner and reasoner can oscillate; use EMA for stability
When NOT to Use: Tasks with abundant labeled data (supervised fine-tuning is simpler); domains where pseudo-labels are unreliable.
Integration: Works with any VLM backbone (CLIP, LLaVA, GPT-4V-style models); no architectural changes needed.
Reference: https://arxiv.org/abs/2511.15661