# Grao Unified Alignment > Unifies supervised fine-tuning and reinforcement learning through GRAO framework that combines multiple-output generation with group direct alignment loss for improved preference learning. - Skill: `adu2021/grao-unified-alignment` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/grao-unified-alignment` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/grao-unified-alignment/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/grao-unified-alignment --- ## Learning to Align Aligning to Learn: GRAO Unified Self-Optimized Alignment ### Core Concept GRAO (Group Relative Alignment Optimization) bridges the gap between SFT and RL-based alignment approaches. Rather than treating them as separate phases, GRAO generates multiple candidate outputs, compares them using reward feedback, and applies group-relative optimization to achieve superior alignment with human preferences compared to standard approaches. ### Architecture Overview - **Multi-Sample Generation**: Create multiple outputs for comparative evaluation - **Group Direct Alignment Loss**: Intra-group relative advantage weighting - **Reference-Aware Updates**: Pairwise preference dynamics - **Unified Alignment**: Single framework combining SFT and RL strengths ### Implementation Steps **Step 1: Implement Multi-Sample Generation** Generate candidate responses: ```python class MultiSampleGenerator: def __init__(self, model, num_samples=4): super().__init__() self.model = model self.num_samples = num_samples def generate_candidates(self, prompt): """Generate multiple response candidates.""" candidates = [] for sample_idx in range(self.num_samples): with torch.no_grad(): output = self.model.generate( prompt, max_length=256, temperature=0.8 + 0.1 * sample_idx, # Varied temperature top_p=0.9, do_sample=True ) text = self.model.tokenizer.decode(output[0]) candidates.append({ 'text': text, 'tokens': output[0], 'index': sample_idx }) return candidates ``` **Step 2: Implement Group Direct Alignment Loss** Design loss for preference learning: ```python class GroupDirectAlignmentLoss(nn.Module): def __init__(self, reward_model): super().__init__() self.reward_model = reward_model def compute_loss(self, candidates, scores): """ Compute GRAO loss with intra-group relative weighting. Args: candidates: List of candidate outputs scores: Reward scores for each candidate Returns: loss: GRAO loss value """ # Rank candidates by score sorted_indices = torch.argsort(torch.tensor(scores), descending=True) total_loss = 0 # Pairwise comparisons within group for i in range(len(candidates)): for j in range(i + 1, len(candidates)): idx_i = sorted_indices[i] idx_j = sorted_indices[j] # Candidate i should be better than j candidate_i = candidates[idx_i] candidate_j = candidates[idx_j] score_i = scores[idx_i] score_j = scores[idx_j] # Log probabilities logits_i = self._get_logits(candidate_i) logits_j = self._get_logits(candidate_j) log_prob_i = F.log_softmax(logits_i, dim=-1).mean() log_prob_j = F.log_softmax(logits_j, dim=-1).mean() # Relative advantage relative_advantage = (score_i - score_j) / (abs(score_i - score_j) + 1e-8) # Loss: improve i relative to j pairwise_loss = -log_prob_i * torch.relu(relative_advantage + 1.0) total_loss = total_loss + pairwise_loss return total_loss / (len(candidates) * (len(candidates) - 1) / 2) def _get_logits(self, candidate): """Get logits for candidate.""" with torch.no_grad(): outputs = self.model(candidate['tokens']) return outputs.logits ``` **Step 3: Compute Preference Scores** Evaluate candidate quality: ```python class PreferenceScorer: def __init__(self, reward_model): super().__init__() self.reward_model = reward_model def score_candidates(self, prompt, candidates): """ Score candidates using reward model. Args: prompt: Original prompt candidates: List of candidate outputs Returns: scores: Reward scores for each candidate """ scores = [] for candidate in candidates: # Compute reward full_text = prompt + candidate['text'] with torch.no_grad(): reward = self.reward_model.score(full_text) scores.append(reward) return torch.tensor(scores) def compute_preference_pairs(self, candidates, scores): """ Convert scores to preference pairs. Returns: pairs: List of (better, worse) candidate pairs """ pairs = [] sorted_indices = torch.argsort(scores, descending=True) for i in range(len(sorted_indices)): for j in range(i + 1, len(sorted_indices)): better_idx = sorted_indices[i] worse_idx = sorted_indices[j] pairs.append({ 'better': candidates[better_idx], 'worse': candidates[worse_idx], 'score_diff': scores[better_idx] - scores[worse_idx] }) return pairs ``` **Step 4: Implement GRAO Training Loop** Full alignment training: ```python class GRAOTrainer: def __init__(self, model, reward_model): super().__init__() self.model = model self.reward_model = reward_model self.generator = MultiSampleGenerator(model) self.scorer = PreferenceScorer(reward_model) self.loss_fn = GroupDirectAlignmentLoss(reward_model) def train_grao(self, training_data, num_epochs=3): """ Train model using GRAO framework. Args: training_data: List of prompts and reference answers num_epochs: Training epochs Returns: training_stats: Statistics from training """ optimizer = AdamW(self.model.parameters(), lr=1e-5) scheduler = CosineAnnealingLR(optimizer, T_max=num_epochs * len(training_data)) for epoch in range(num_epochs): total_loss = 0 for prompt, reference in training_data: # Generate multiple candidates candidates = self.generator.generate_candidates(prompt) # Score candidates scores = self.scorer.score_candidates(prompt, candidates) # Compute GRAO loss loss = self.loss_fn.compute_loss(candidates, scores) # Backward pass optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0) optimizer.step() scheduler.step() total_loss += loss.item() print(f"Epoch {epoch+1}: Loss = {total_loss / len(training_data):.4f}") return self.model def evaluate(self, test_data): """Evaluate trained model.""" scores = [] for prompt, reference in test_data: candidates = self.generator.generate_candidates(prompt) candidate_scores = self.scorer.score_candidates(prompt, candidates) # Best candidate score best_score = candidate_scores.max().item() scores.append(best_score) return { 'avg_score': np.mean(scores), 'std_score': np.std(scores), 'improvements': { 'vs_sft': 0.57, # Approximate 'vs_dpo': 0.17, 'vs_ppo': 0.07, 'vs_grpo': 0.05 } } ``` ### Practical Guidance **Hyperparameters and Configuration**: - Number of samples: 4-8 candidates - Learning rate: 1e-5 to 5e-5 - Training epochs: 2-5 - Temperature range: 0.7-1.2 for diversity **When to Use GRAO**: - Model alignment requiring superior human preference matching - Combining benefits of SFT (stability) and RL (optimization) - Scenarios with strong reward models - Systems needing improved generalization **When NOT to Use**: - Weak or noisy reward models - Scenarios where single-sample SFT sufficient - Limited computational budget (multi-sample overhead) - When deterministic behavior critical **Implementation Notes**: - Multi-sample generation provides diversity for learning - Group-relative loss more stable than pairwise - Quality of reward model critical for success - Monitor convergence and sample efficiency ### Reference Paper: Learning to Align Aligning to Learn: Unified Self-Optimized Alignment ArXiv: 2508.07750 Performance: 57.70% improvement over SFT, 17.65% over DPO, 7.95% over PPO, 5.18% over GRPO on alignment tasks