Run Rlhf Code Experiment

Run RLHF Code Experiment

Use this skill when the user wants to run, adapt, compare, or document an experiment from code/.

Pick The Starting Point

• Instruction Fine-Tuning: read code/instruction_tuning/README.md.
• Policy gradients / RL / GRPO / PPO: read code/policy_gradients/README.md.
• Reward models / ORM / PRM / Bradley-Terry RM: read code/reward_models/README.md.
• DPO / IPO / SimPO / ORPO / KTO / APO: read code/direct_alignment/README.md.
• Rejection sampling / best-of-N / GSM8K filtering: read code/rejection_sampling/README.md.
• Distillation: read code/distillation/README.md.

Run Protocol

1. Work from the repository root unless a command explicitly says cd code/.
2. Install or refresh dependencies with cd code/ && uv sync only when needed.
3. Use uv run python, never bare python.
4. Start with a short run:
   • Reward models: lower --samples and --epochs.
   • Direct alignment: use --max_samples or copy a YAML with a smaller sample count.
   • Policy gradients: copy a YAML and reduce data.size before changing algorithm logic.
   • Rejection sampling: reduce max_train_samples, max_test_samples, or num_completions_per_prompt in a copied YAML.
5. For any long training, preprocessing, evaluation, or sweep command, launch the command in the background rather than the foreground. In Claude Code, use the background-run option for the shell command, then start a monitor for it.
6. Watch the monitor until the run has produced initial logs or failed. The Claude Code status bar should show a background task and monitor (for example, [1 background task] [1 monitor]). Keep checking the monitor periodically for loss, metrics, W&B URLs, OOMs, dataset download errors, and stalled output.
7. Run one training job at a time unless GPU memory has been checked.
8. If W&B is not desired, set WANDB_MODE=disabled or use the module's no-W&B flag when available.

What To Report

Report enough detail for another reader to reproduce the result:

• Exact command.
• Model, dataset, seed, and config file.
• Config values changed from the checked-in defaults.
• Final metrics and any observed failure mode.
• W&B run URL if logging was enabled.
• Follow-up sweep worth trying next.

Comparison Rules

• For policy gradients, compare avg_correctness, avg_format, avg_binary, loss, and whether sampled groups contain reward contrast.
• For reward models, compare reward margins or correctness scores on held-out examples, not just training loss.
• For direct alignment, compare accuracy, margins, chosen_rewards, rejected_rewards, and sample generations. IPO loss scale is not directly comparable to DPO loss scale.
• For rejection sampling, always compare each reward-selected run to its matched random baseline.

Documentation Rule

If the run exposes a new setup requirement, failure mode, or useful workflow shortcut, update the relevant README, code/CLAUDE.md, or this skill before finishing.