# Run Rlhf Code Experiment

> Plan, run, and report a small RLHF Book code experiment.

- Skill: `natolambert/run-rlhf-code-experiment` (Agent Skill)
- Install (CLI): `npx skillmds@latest add natolambert/run-rlhf-code-experiment`
- Raw SKILL.md: https://api.skillmd.com/api/skills/natolambert/run-rlhf-code-experiment/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: natolambert (https://skillmd.com/u/natolambert)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/natolambert/run-rlhf-code-experiment

---


# Run RLHF Code Experiment

Use this skill when the user wants to run, adapt, compare, or document an experiment from `code/`.

## Pick The Starting Point

- Instruction Fine-Tuning: read `code/instruction_tuning/README.md`.
- Policy gradients / RL / GRPO / PPO: read `code/policy_gradients/README.md`.
- Reward models / ORM / PRM / Bradley-Terry RM: read `code/reward_models/README.md`.
- DPO / IPO / SimPO / ORPO / KTO / APO: read `code/direct_alignment/README.md`.
- Rejection sampling / best-of-N / GSM8K filtering: read `code/rejection_sampling/README.md`.
- Distillation: read `code/distillation/README.md`.

## Run Protocol

1. Work from the repository root unless a command explicitly says `cd code/`.
2. Install or refresh dependencies with `cd code/ && uv sync` only when needed.
3. Use `uv run python`, never bare `python`.
4. Start with a short run:
   - Reward models: lower `--samples` and `--epochs`.
   - Direct alignment: use `--max_samples` or copy a YAML with a smaller sample count.
   - Policy gradients: copy a YAML and reduce `data.size` before changing algorithm logic.
   - Rejection sampling: reduce `max_train_samples`, `max_test_samples`, or `num_completions_per_prompt` in a copied YAML.
5. For any long training, preprocessing, evaluation, or sweep command, launch the command in the background rather than the foreground. In Claude Code, use the background-run option for the shell command, then start a monitor for it.
6. Watch the monitor until the run has produced initial logs or failed. The Claude Code status bar should show a background task and monitor (for example, `[1 background task] [1 monitor]`). Keep checking the monitor periodically for loss, metrics, W&B URLs, OOMs, dataset download errors, and stalled output.
7. Run one training job at a time unless GPU memory has been checked.
8. If W&B is not desired, set `WANDB_MODE=disabled` or use the module's no-W&B flag when available.

## What To Report

Report enough detail for another reader to reproduce the result:

- Exact command.
- Model, dataset, seed, and config file.
- Config values changed from the checked-in defaults.
- Final metrics and any observed failure mode.
- W&B run URL if logging was enabled.
- Follow-up sweep worth trying next.

## Comparison Rules

- For policy gradients, compare `avg_correctness`, `avg_format`, `avg_binary`, loss, and whether sampled groups contain reward contrast.
- For reward models, compare reward margins or correctness scores on held-out examples, not just training loss.
- For direct alignment, compare `accuracy`, `margins`, `chosen_rewards`, `rejected_rewards`, and sample generations. IPO loss scale is not directly comparable to DPO loss scale.
- For rejection sampling, always compare each reward-selected run to its matched random baseline.

## Documentation Rule

If the run exposes a new setup requirement, failure mode, or useful workflow shortcut, update the relevant README, `code/CLAUDE.md`, or this skill before finishing.