Debugging RL Training
When To Use
Use this skill when a user needs to debug learned behavior, reward hacking, checkpoint compatibility, unstable training, or training-result trustworthiness.
Do not use this skill for first-time training commands. Use isaaclab-training-rl-agents for launch commands and agent config wiring.
Workflow
- Identify task name, workflow type, RL library, agent config, seed, backend, and exact launch command.
- Confirm the environment contract: action space, observation space, reward terms, termination terms, reset logic, and success metric.
- Run the smallest reproduction: import, reset/step, one-iteration training, or deterministic playback depending on where the failure appears.
- Change one variable per training experiment. Mark multi-variable runs as exploratory.
- Compare reward curves against task metrics. Reward increases are not proof that the task behavior improved.
- For reward issues, map every reward term to a named task phase and check that success reward, termination, and evaluation metric use consistent geometry.
- For checkpoint issues, compare current observation/action dimensions with the saved training configuration before editing policy code.
- For contact-rich tasks, collect state traces for controlled-frame pose, object pose, contacts, gripper state, per-term rewards, and termination flags.
- Select checkpoints by task metrics, rollout behavior, and stability, not reward alone.
Validation
Use this checklist:
- The exact command and failing symptom are recorded.
- The failed layer is classified as environment, reward, reset, physics, runner, or checkpoint compatibility.
- A focused reproduction isolates one variable.
- Reward terms and task metrics are inspected together.
- A deterministic rollout or state trace confirms the behavior change.
- Any recommended next run changes only one variable.
For skill changes, run:
uv run --no-project python tools/skills/cli.py check
Maintenance
Keep this skill synchronized with skills/user/train-rl-agents/, docs/source/concepts/reinforcement_learning.rst, the uv-based train and play entry points, and task examples under source/isaaclab_tasks/isaaclab_tasks/. If recurring reward or checkpoint guidance belongs in user docs, update docs/source/ first.
References
1---2name: isaaclab-debugging-rl-training3description: Diagnoses Isaac Lab reinforcement learning behavior, rewards, metrics, checkpoints, and training experiments. Use when reward curves look wrong, policies fail despite training, checkpoints mismatch, or RL changes need focused ablations.4audience: user5status: experimental6owners:7 - isaaclab-maintainers8metadata:9 source: https://skillmd.com/skills/isaac-sim/isaaclab-debugging-rl-training10---1112# Debugging RL Training1314## When To Use1516Use this skill when a user needs to debug learned behavior, reward hacking, checkpoint compatibility, unstable training, or training-result trustworthiness.1718Do not use this skill for first-time training commands. Use `isaaclab-training-rl-agents` for launch commands and agent config wiring.1920## Workflow21221. Identify task name, workflow type, RL library, agent config, seed, backend, and exact launch command.232. Confirm the environment contract: action space, observation space, reward terms, termination terms, reset logic, and success metric.243. Run the smallest reproduction: import, reset/step, one-iteration training, or deterministic playback depending on where the failure appears.254. Change one variable per training experiment. Mark multi-variable runs as exploratory.265. Compare reward curves against task metrics. Reward increases are not proof that the task behavior improved.276. For reward issues, map every reward term to a named task phase and check that success reward, termination, and evaluation metric use consistent geometry.287. For checkpoint issues, compare current observation/action dimensions with the saved training configuration before editing policy code.298. For contact-rich tasks, collect state traces for controlled-frame pose, object pose, contacts, gripper state, per-term rewards, and termination flags.309. Select checkpoints by task metrics, rollout behavior, and stability, not reward alone.3132## Validation3334Use this checklist:35361. The exact command and failing symptom are recorded.372. The failed layer is classified as environment, reward, reset, physics, runner, or checkpoint compatibility.383. A focused reproduction isolates one variable.394. Reward terms and task metrics are inspected together.405. A deterministic rollout or state trace confirms the behavior change.416. Any recommended next run changes only one variable.4243For skill changes, run:4445```bash46uv run --no-project python tools/skills/cli.py check47```4849## Maintenance5051Keep this skill synchronized with `skills/user/train-rl-agents/`, `docs/source/concepts/reinforcement_learning.rst`, the uv-based `train` and `play` entry points, and task examples under `source/isaaclab_tasks/isaaclab_tasks/`. If recurring reward or checkpoint guidance belongs in user docs, update `docs/source/` first.5253## References5455- [Reference](reference.md)56- [Examples](examples.md)57- [Evaluations](evaluations.md)58- [RL training skill](../train-rl-agents/SKILL.md)59- [RL training guide](../../../docs/source/concepts/reinforcement_learning.rst)60- [Task examples](../../../source/isaaclab_tasks/isaaclab_tasks)