Rlinf Benchmarking

Skill 14: Benchmarking

Purpose

Evaluate Physical AI RL training quality and infrastructure performance using standardized benchmarks. This covers both task success rate (model quality) and training throughput (infrastructure efficiency).

Benchmarks

LIBERO

LIBERO is the primary benchmark for RLinf. It evaluates lifelong robot learning across diverse manipulation tasks.

RoboTwin 2.0

Bimanual manipulation with domain randomization:

Running Benchmarks on EKS

LIBERO Evaluation

# eval-libero.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: eval-libero-long
spec:
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        role: gpu-training
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: eval
          image: <ECR_URI>:latest
          command:
            - bash
            - -c
            - |
              # Point to the RL-trained checkpoint
              MODEL_PATH="/fsx/checkpoints/libero10-rl/actor/global_step_100"

              bash examples/overwrite_vla_ckpt_utils.sh $MODEL_PATH

              HYDRA_FULL_ERROR=1 python -u -m verl.trainer.main_ppo \
                data.task_suite_name=libero_10 \
                data.val_batch_size=496 \
                actor_rollout_ref.model.path=$MODEL_PATH \
                actor_rollout_ref.model.vla=openvla-oft \
                actor_rollout_ref.model.action_token_len=7 \
                actor_rollout_ref.model.action_chunks_len=8 \
                actor_rollout_ref.rollout.temperature=1.0 \
                actor_rollout_ref.rollout.task_suite_name=libero_10 \
                actor_rollout_ref.rollout.pretrained_checkpoint=$MODEL_PATH \
                trainer.val_only=True \
                trainer.val_before_train=True \
                trainer.n_gpus_per_node=8 \
                trainer.nnodes=1
          resources:
            requests:
              nvidia.com/gpu: 8
              vpc.amazonaws.com/efa: 4
            limits:
              nvidia.com/gpu: 8
              vpc.amazonaws.com/efa: 4
          volumeMounts:
            - name: fsx
              mountPath: /fsx
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: fsx
          persistentVolumeClaim:
            claimName: fsx-claim
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "64Gi"

Evaluation Protocol

RLinf evaluation uses:

• Greedy decoding (do_sample=False, temperature=1.0 with greedy)
• 50 held-out test scenarios per task
• 3 evaluation runs for reproducibility
• Success Rate (SR) as the primary metric

Reference Results

LIBERO (from the paper)

RoboTwin 2.0 (from the paper)

Infrastructure Performance Benchmarks

GPU Utilization

Track GPU utilization during training to identify bottlenecks:

# In training pod
watch -n 1 nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used,memory.total --format=csv

Target:

• GPU utilization: >70% during policy update, lower during rollout (expected)
• Memory utilization: >80% (efficient VRAM usage)

NCCL AllReduce Bandwidth

# Run nccl-tests between nodes
/usr/bin/all_reduce_perf -b 8 -e 1G -f 2 -g 8

Training Throughput

Measure end-to-end training speed:

Storage I/O

# Sequential write throughput (checkpoint simulation)
dd if=/dev/zero of=/fsx/benchmark_write bs=1M count=4096 oflag=direct
# Expected: >500 MB/s for 4.8 TiB filesystem at 125 MB/s/TiB

# Sequential read throughput (model loading simulation)
dd if=/fsx/benchmark_write of=/dev/null bs=1M count=4096 iflag=direct

Comparing EKS vs Paper Results

When reproducing paper results on EKS, expect:

Success rates should be equivalent across hardware. Training speed depends on GPU generation and interconnect.

Benchmark Automation

Scheduled Evaluation

Run evaluation automatically after each checkpoint save:

# In a CronJob or triggered by a webhook
for step in 25 50 75 100; do
  CKPT="/fsx/checkpoints/${EXPERIMENT}/actor/global_step_${step}"
  if [ -d "$CKPT" ]; then
    kubectl create job eval-step-${step} --from=job/eval-libero-long \
      -- --env MODEL_PATH=$CKPT
  fi
done

Results Collection

Evaluation results are logged to MLflow. Extract programmatically:

import mlflow
from mlflow.tracking import MlflowClient

client = MlflowClient("http://mlflow.mlflow.svc.cluster.local:5000")
experiment = client.get_experiment_by_name("RLinf")
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    filter_string="params.val_only = 'True'"
)
for run in runs:
    metrics = run.data.metrics
    print(f"{run.info.run_name}: SR={metrics.get('success_rate', 'N/A')}")

Validation Checklist

• LIBERO evaluation runs and produces per-task success rates
• Success rates are in expected range (compare to paper)
• RoboTwin evaluation runs (if applicable)
• GPU utilization > 70% during policy updates
• NCCL bandwidth matches expectations for instance type
• Checkpoint save/load time is acceptable (< 60s)
• Results are logged to MLflow for comparison

Related Skills

• Skill 08: PyTorch Training Scripts - Training configuration
• Skill 13: Functional Testing - Pre-benchmark validation
• Skill 15: Monitoring and Observability - GPU/training metrics