Slime Preference Optimization

SLIME: Stabilized Preference Optimization

Problem

Margin-based preference optimization methods like DPO and SimPO can degrade the quality of preferred responses while optimizing margins. Over-aggressive suppression of rejected tokens removes valid syntax and reasoning patterns.

A reference-free method that preserves preferred response quality while improving alignment is needed.

Core Concept

SLIME decouples preference learning from generation quality through three components: explicit likelihood anchoring of chosen responses, token-level softplus stabilization preventing probability collapse, and dual-margin optimization with soft and hard constraints.

This ensures the model maintains high probability for preferred continuations while treating rejected responses with measured regularization rather than complete suppression.

Architecture Overview

• Likelihood Anchoring: Maximize log-probability of chosen responses explicitly
• Token-Level Stabilization: Softplus penalty prevents rejected probability collapse
• Hard Margin: Defines victory condition where loss becomes zero
• Soft Margin: Continued gradient signal beyond hard margin
• Reference-Free: No need for reference model like in DPO
• Dual-Margin Design: Balances margin satisfaction with output quality

Implementation

Step 1: Define SLIME Loss Function

Implement anchored likelihood with stabilized margins.

import torch
import torch.nn.functional as F

def slime_loss(chosen_logits, rejected_logits, hard_margin=1.0, soft_margin=0.5):
    """Compute SLIME loss combining likelihood and margin objectives."""
    # Get probabilities
    chosen_probs = F.softmax(chosen_logits, dim=-1)
    rejected_probs = F.softmax(rejected_logits, dim=-1)

    # Component 1: Likelihood anchoring—maximize chosen probability
    # Higher is better, so negative for loss
    chosen_log_prob = torch.log(chosen_probs + 1e-8)
    likelihood_loss = -chosen_log_prob.mean()

    # Component 2: Margin constraint with dual margins
    margin = chosen_probs - rejected_probs

    # Hard margin: loss = 0 when margin >= hard_margin
    hard_margin_loss = F.relu(hard_margin - margin).pow(2).mean()

    # Soft margin: additional penalty beyond hard margin (continued learning)
    soft_margin_penalty = F.softplus(soft_margin - margin).mean()

    # Component 3: Token-level stabilization—prevent rejected collapse
    # Softplus prevents log(rejected_prob) from exploding negatively
    rejected_stability = F.softplus(-torch.log(rejected_probs + 1e-8)).mean()

    # Total loss: balance all components
    total_loss = (0.5 * likelihood_loss +
                  0.3 * hard_margin_loss +
                  0.1 * soft_margin_penalty +
                  0.1 * rejected_stability)

    return total_loss

Step 2: Batch Preference Training

Train on preference pairs using SLIME objective.

def train_with_slime(model, preference_pairs, num_epochs=3, learning_rate=1e-4):
    """Train model on preference pairs using SLIME loss."""
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    for epoch in range(num_epochs):
        total_loss = 0

        for batch in create_batches(preference_pairs, batch_size=32):
            losses = []

            for chosen, rejected in batch:
                # Tokenize inputs
                chosen_tokens = model.tokenize(chosen)
                rejected_tokens = model.tokenize(rejected)

                # Get logits
                chosen_logits = model.get_logits(chosen_tokens[:-1])
                rejected_logits = model.get_logits(rejected_tokens[:-1])

                # Target tokens for next position
                chosen_targets = chosen_tokens[1:]
                rejected_targets = rejected_tokens[1:]

                # Extract logits for target positions
                chosen_target_logits = chosen_logits[torch.arange(len(chosen_targets)), chosen_targets]
                rejected_target_logits = rejected_logits[torch.arange(len(rejected_targets)), rejected_targets]

                # Compute SLIME loss
                loss = slime_loss(chosen_target_logits, rejected_target_logits)
                losses.append(loss)

            # Batch gradient update
            batch_loss = torch.stack(losses).mean()
            batch_loss.backward()
            optimizer.step()
            optimizer.zero_grad()

            total_loss += batch_loss.item()

        print(f"Epoch {epoch}, Average Loss: {total_loss / len(batch)}")

    return model

Step 3: Token-Level Stabilization Module

Implement softplus-based stabilization for rejected responses.

class StabilizedMarginOptimizer:
    def __init__(self, hard_margin=1.0, soft_margin=0.5, stability_strength=0.1):
        self.hard_margin = hard_margin
        self.soft_margin = soft_margin
        self.stability_strength = stability_strength

    def compute_stabilization_penalty(self, rejected_logits):
        """Prevent rejected probability from collapsing."""
        rejected_probs = F.softmax(rejected_logits, dim=-1)

        # Log probabilities
        log_rejected = torch.log(rejected_probs + 1e-8)

        # Softplus prevents extreme negative values
        stability_penalty = F.softplus(-log_rejected)

        # Weighted penalty
        total_penalty = self.stability_strength * stability_penalty.mean()

        return total_penalty

    def compute_margin_loss(self, chosen_logits, rejected_logits):
        """Compute dual-margin loss."""
        chosen_probs = F.softmax(chosen_logits, dim=-1)
        rejected_probs = F.softmax(rejected_logits, dim=-1)

        margin = chosen_probs - rejected_probs

        # Hard margin satisfaction
        hard_loss = F.relu(self.hard_margin - margin).pow(2).mean()

        # Soft margin for continued learning
        soft_loss = F.softplus(self.soft_margin - margin).mean()

        return hard_loss + 0.5 * soft_loss

Step 4: Comparison with Baselines

Validate SLIME quality preservation against DPO/SimPO.

def compare_preference_methods(preference_pairs, test_prompts):
    """Compare SLIME against DPO and SimPO."""
    results = {}

    for method in ['slime', 'dpo', 'simpo']:
        model = initialize_model()

        if method == 'slime':
            model = train_with_slime(model, preference_pairs)
        elif method == 'dpo':
            model = train_with_dpo(model, preference_pairs)
        else:
            model = train_with_simpo(model, preference_pairs)

        # Evaluate quality preservation
        preferred_quality = evaluate_response_quality(model, test_prompts)
        alignment = evaluate_preference_alignment(model, preference_pairs)

        results[method] = {
            'quality': preferred_quality,
            'alignment': alignment
        }

    return results

Practical Guidance

Hyperparameter Configuration

When to Use

• Aligning LLMs with human preferences while preserving capability
• Tasks where quality of preferred response is critical
• Scenarios where rejected responses contain valid patterns (code syntax, reasoning steps)
• Reference-free alignment (no external reference model available)
• Multi-model comparison on alignment leaderboards (MT-Bench, Arena)

When Not to Use

• When preferred responses are inherently low-quality (optimization cannot fix)
• Tasks where strong margin separation is sole objective
• Models already well-aligned (further optimization not needed)
• Computational constraints requiring simpler methods
• Scenarios where reference models are accurate and available

Common Pitfalls

1. Margin miscalibration: Margins too tight create instability; too loose lose learning signal. Start with default, monitor.
2. Stability-alignment trade-off: High stability strength may prevent good margin learning. Balance via validation set.
3. Quality degradation: SLIME prevents it better than baselines, but monitor preferred response outputs for subtle decline.
4. Batch size sensitivity: Small batches may increase variance. Use batch size 16+.

Reference

SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization https://arxiv.org/abs/2602.02383