Rlinf

rlinf

Operational guide for running RLinf (Ray-based RL infrastructure) on UCR HPCC, covering common issues, debugging patterns, and configuration pitfalls.

Environment Setup

Critical Debugging Patterns

1. Demo Buffer Format (trajectory_index.json)

RLinf's TrajectoryReplayBuffer.load_checkpoint() expects a specific JSON structure. Getting this wrong causes silent infinite busy-wait (no error, no crash, just hangs).

Correct format:

{
  "trajectory_index": {
    "0": {
      "num_samples": 16,
      "trajectory_id": 0,
      "max_episode_length": 50,
      "shape": [2, 8],
      "model_weights_id": "demo_uuid_0",
      "filename": "trajectory_0_demo_uuid_0.pt"
    }
  },
  "trajectory_id_list": [0, 1, 2, ...]
}

Wrong format (causes hang):

{
  "0": {"model_weights_id": "...", "filename": "..."}
}

Why it hangs: load_checkpoint() looks for index_data.get("trajectory_index", {}) and index_data.get("trajectory_id_list", []). If these keys are missing, the buffer loads with size=0, and ReplayBufferDataset.__iter__() spins forever in a busy-wait loop checking is_ready() which always returns False.

metadata.json must include:

{
  "trajectory_format": "pt",
  "size": <num_trajectories>,
  "total_samples": <total>,
  "trajectory_counter": <num_trajectories>,
  "seed": 1234
}

The "size" key is critical — without it, metadata.get("size", 0) returns 0.

2. Hydra Config Overrides

Pitfall: Combined placement keys like actor,env,rollout: 0 in YAML cannot be partially overridden via CLI. Split them into separate keys in the base YAML first.

3. Script Safety with set -euo pipefail

Glob failures in ls -d with no matches return exit code 1, which set -e treats as fatal. Always append || true:

LATEST=$(ls -d ${DIR}/global_step_* 2>/dev/null | sort -n | tail -1 || true)

5. ManiSkill Sim Backend

• Must use sim_backend: "gpu" — CPU mode doesn't support vectorized envs
• Vulkan required — cannot run on login nodes (no GPU)
• Demo conversion must run on GPU nodes via sbatch

RLPD-Specific Notes

RLPD (RL with Prior Data) adds a demo_buffer alongside the replay_buffer:

The ReplayBufferDataset mixes 50/50 from replay and demo buffer when demo_buffer is provided.

Debugging Stuck Training

1. Check stderr log line count over time — if not growing, job is hung
2. Look for "Replay buffer size X < Y, skipping training" — buffer not filling
3. Check demo_buffer format — most common cause of silent hang
4. Check tensorboard event file size — growing = training active
5. Don't assume GPU contention — multiple jobs on same node work fine if each gets its own GPU via SLURM --gres

Auto-Resubmit Pattern

EXIT_CODE=0
python train_embodied_agent.py ... || EXIT_CODE=$?

if [ $EXIT_CODE -eq 143 ]; then
    # SIGTERM from SLURM time limit
    sbatch ... "$SCRIPT_PATH" "$ARGS"
fi

Exit code 143 = SIGTERM (128 + 15), sent by SLURM when time limit reached.