# Optimization Grpo

> GRPO/RL fine-tuning for DSPy programs

- Skill: `j33bs/optimization-grpo` (Agent Skill)
- Install (CLI): `npx skillmds@latest add j33bs/optimization-grpo`
- Raw SKILL.md: https://api.skillmd.com/api/skills/j33bs/optimization-grpo/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: j33bs (https://skillmd.com/u/j33bs)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/j33bs/optimization-grpo

---


# GRPO/RL Fine-Tuning

## 🎯 Trigger Conditions
Use when asked about GRPO (Group Relative Policy Optimization), RL fine-tuning for LLMs, or reinforcement learning for DSPy.

## 📚 Prerequisites
- `trl` package installed
- `transformers` package installed
- GPU available (recommended)
- Training data prepared

## 🛠️ GRPO Implementation

### 1. Basic GRPO Setup
```python
from trl import GRPOConfig, GRPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("your-model")
tokenizer = AutoTokenizer.from_pretrained("your-model")

# Configure GRPO
config = GRPOConfig(
    learning_rate=1e-5,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    output_dir="grpo-output"
)

# Create trainer
trainer = GRPOTrainer(
    model=model,
    config=config,
    train_dataset=your_dataset,
    tokenizer=tokenizer
)

# Train
trainer.train()
```

### 2. GRPO with Reward Model
```python
from trl import GRPOConfig, GRPOTrainer

# Define reward function
def reward_fn(completions, **kwargs):
    rewards = []
    for completion in completions:
        # Calculate reward based on quality
        reward = calculate_quality(completion)
        rewards.append(reward)
    return rewards

# Create trainer with reward
trainer = GRPOTrainer(
    model=model,
    config=config,
    train_dataset=your_dataset,
    tokenizer=tokenizer,
    reward_fn=reward_fn
)
```

### 3. GRPO for DSPy
```python
import dspy

# Create DSPy program
program = dspy.ChainOfThought("question -> reasoning -> answer")

# Prepare training data
train_data = [
    dspy.Example(
        question="What is X?",
        answer="Y"
    ).with_inputs("question")
]

# Fine-tune with GRPO
trainer = GRPOTrainer(
    model=program.model,
    config=grpo_config,
    train_dataset=train_data,
    reward_fn=your_reward_function
)

trainer.train()
```

### 4. GRPO Hyperparameter Tuning
```python
config = GRPOConfig(
    # Learning rate schedule
    learning_rate=1e-5,
    lr_scheduler_type="cosine",
    
    # Training parameters
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    
    # GRPO-specific
    beta=0.1,  # KL divergence coefficient
    gamma=0.99,  # Reward discount factor
    
    # Regularization
    weight_decay=0.01,
    max_grad_norm=1.0
)
```

## ⚠️ Pitfalls
- **Reward design**: Poor rewards lead to poor policies
- **Compute cost**: GRPO requires significant resources
- **Stability**: RL can be unstable
- **Overfitting**: Monitor validation performance

## 📖 References
- [TRL GRPO](https://huggingface.co/docs/trl/main/en/grpo_trainer)
- [GRPO Paper](https://arxiv.org/abs/2503.18892)