adu2021 Learning to Align Aligning to Learn: GRAO Unified Self-Optimized Alignment
Core Concept
GRAO (Group Relative Alignment Optimization) bridges the gap between SFT and RL-based alignment approaches. Rather than treating them as separate phases, GRAO generates multiple candidate outputs, compares them using reward feedback, and applies group-relative optimization to achieve superior alignment with human preferences compared to standard approaches.
Architecture Overview
- Multi-Sample Generation: Create multiple outputs for comparative evaluation
- Group Direct Alignment Loss: Intra-group relative advantage weighting
- Reference-Aware Updates: Pairwise preference dynamics
- Unified Alignment: Single framework combining SFT and RL strengths
Implementation Steps
Step 1: Implement Multi-Sample Generation
Generate candidate responses:
class MultiSampleGenerator:
def __init__(self, model, num_samples=4):
super().__init__()
self.model = model
self.num_samples = num_samples
def generate_candidates(self, prompt):
"""Generate multiple response candidates."""
candidates = []
for sample_idx in range(self.num_samples):
with torch.no_grad():
output = self.model.generate(
prompt,
max_length=256,
temperature=0.8 + 0.1 * sample_idx, # Varied temperature
top_p=0.9,
do_sample=True
)
text = self.model.tokenizer.decode(output[0])
candidates.append({
'text': text,
'tokens': output[0],
'index': sample_idx
})
return candidates
Step 2: Implement Group Direct Alignment Loss
Design loss for preference learning:
class GroupDirectAlignmentLoss(nn.Module):
def __init__(self, reward_model):
super().__init__()
self.reward_model = reward_model
def compute_loss(self, candidates, scores):
"""
Compute GRAO loss with intra-group relative weighting.
Args:
candidates: List of candidate outputs
scores: Reward scores for each candidate
Returns:
loss: GRAO loss value
"""
# Rank candidates by score
sorted_indices = torch.argsort(torch.tensor(scores), descending=True)
total_loss = 0
# Pairwise comparisons within group
for i in range(len(candidates)):
for j in range(i + 1, len(candidates)):
idx_i = sorted_indices[i]
idx_j = sorted_indices[j]
# Candidate i should be better than j
candidate_i = candidates[idx_i]
candidate_j = candidates[idx_j]
score_i = scores[idx_i]
score_j = scores[idx_j]
# Log probabilities
logits_i = self._get_logits(candidate_i)
logits_j = self._get_logits(candidate_j)
log_prob_i = F.log_softmax(logits_i, dim=-1).mean()
log_prob_j = F.log_softmax(logits_j, dim=-1).mean()
# Relative advantage
relative_advantage = (score_i - score_j) / (abs(score_i - score_j) + 1e-8)
# Loss: improve i relative to j
pairwise_loss = -log_prob_i * torch.relu(relative_advantage + 1.0)
total_loss = total_loss + pairwise_loss
return total_loss / (len(candidates) * (len(candidates) - 1) / 2)
def _get_logits(self, candidate):
"""Get logits for candidate."""
with torch.no_grad():
outputs = self.model(candidate['tokens'])
return outputs.logits
Step 3: Compute Preference Scores
Evaluate candidate quality:
class PreferenceScorer:
def __init__(self, reward_model):
super().__init__()
self.reward_model = reward_model
def score_candidates(self, prompt, candidates):
"""
Score candidates using reward model.
Args:
prompt: Original prompt
candidates: List of candidate outputs
Returns:
scores: Reward scores for each candidate
"""
scores = []
for candidate in candidates:
# Compute reward
full_text = prompt + candidate['text']
with torch.no_grad():
reward = self.reward_model.score(full_text)
scores.append(reward)
return torch.tensor(scores)
def compute_preference_pairs(self, candidates, scores):
"""
Convert scores to preference pairs.
Returns:
pairs: List of (better, worse) candidate pairs
"""
pairs = []
sorted_indices = torch.argsort(scores, descending=True)
for i in range(len(sorted_indices)):
for j in range(i + 1, len(sorted_indices)):
better_idx = sorted_indices[i]
worse_idx = sorted_indices[j]
pairs.append({
'better': candidates[better_idx],
'worse': candidates[worse_idx],
'score_diff': scores[better_idx] - scores[worse_idx]
})
return pairs
Step 4: Implement GRAO Training Loop
Full alignment training:
class GRAOTrainer:
def __init__(self, model, reward_model):
super().__init__()
self.model = model
self.reward_model = reward_model
self.generator = MultiSampleGenerator(model)
self.scorer = PreferenceScorer(reward_model)
self.loss_fn = GroupDirectAlignmentLoss(reward_model)
def train_grao(self, training_data, num_epochs=3):
"""
Train model using GRAO framework.
Args:
training_data: List of prompts and reference answers
num_epochs: Training epochs
Returns:
training_stats: Statistics from training
"""
optimizer = AdamW(self.model.parameters(), lr=1e-5)
scheduler = CosineAnnealingLR(optimizer, T_max=num_epochs * len(training_data))
for epoch in range(num_epochs):
total_loss = 0
for prompt, reference in training_data:
# Generate multiple candidates
candidates = self.generator.generate_candidates(prompt)
# Score candidates
scores = self.scorer.score_candidates(prompt, candidates)
# Compute GRAO loss
loss = self.loss_fn.compute_loss(candidates, scores)
# Backward pass
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
optimizer.step()
scheduler.step()
total_loss += loss.item()
print(f"Epoch {epoch+1}: Loss = {total_loss / len(training_data):.4f}")
return self.model
def evaluate(self, test_data):
"""Evaluate trained model."""
scores = []
for prompt, reference in test_data:
candidates = self.generator.generate_candidates(prompt)
candidate_scores = self.scorer.score_candidates(prompt, candidates)
# Best candidate score
best_score = candidate_scores.max().item()
scores.append(best_score)
return {
'avg_score': np.mean(scores),
'std_score': np.std(scores),
'improvements': {
'vs_sft': 0.57, # Approximate
'vs_dpo': 0.17,
'vs_ppo': 0.07,
'vs_grpo': 0.05
}
}
Practical Guidance
Hyperparameters and Configuration:
- Number of samples: 4-8 candidates
- Learning rate: 1e-5 to 5e-5
- Training epochs: 2-5
- Temperature range: 0.7-1.2 for diversity
When to Use GRAO:
- Model alignment requiring superior human preference matching
- Combining benefits of SFT (stability) and RL (optimization)
- Scenarios with strong reward models
- Systems needing improved generalization
When NOT to Use:
- Weak or noisy reward models
- Scenarios where single-sample SFT sufficient
- Limited computational budget (multi-sample overhead)
- When deterministic behavior critical
Implementation Notes:
- Multi-sample generation provides diversity for learning
- Group-relative loss more stable than pairwise
- Quality of reward model critical for success
- Monitor convergence and sample efficiency
Reference
Paper: Learning to Align Aligning to Learn: Unified Self-Optimized Alignment ArXiv: 2508.07750 Performance: 57.70% improvement over SFT, 17.65% over DPO, 7.95% over PPO, 5.18% over GRPO on alignment tasks