Functional Testing

Skill 13: Functional Testing

Purpose

Validate the RL training stack end-to-end before launching production training runs. This includes smoke tests for individual components, integration tests for the full pipeline, and regression checks after infrastructure or code changes.

RLinf Example: See the rlinf-functional-testing skill for RLinf-specific L5 test configurations and expected outputs.

Testing Levels

Level 0: Infrastructure  -->  GPU, EFA, storage accessible
Level 1: Runtime          -->  PyTorch, CUDA, libraries working
Level 2: Single-GPU       -->  Model loads, forward pass succeeds
Level 3: Multi-GPU        -->  FSDP wrapping, NCCL allreduce
Level 3.5: EFA Validation -->  NCCL over EFA (Phase 0, cheap instances)
Level 4: Rollout          -->  Simulation env renders, actions generated
Level 5: End-to-End       -->  Full training step completes (rollout --> update)
Level 6: Validation       -->  Evaluation produces success rate metrics

Phase 0: EFA Validation Sprint (MUST DO FIRST)

CRITICAL: Always validate EFA/NCCL on cheap instances (g6.8xlarge, ~$1/hr) before purchasing expensive capacity blocks (p5.48xlarge, ~$725/hr). A failed $1,450 capacity block taught this lesson.

Prerequisites

1. EKS cluster with GPU nodes (2x g6.8xlarge for cheapest test)
2. NVIDIA GPU Operator running
3. EFA device plugin running
4. Kubeflow MPI Operator installed (enable_mpi_operator = true in terraform.tfvars)

Build NCCL Test Image

A pre-built image is available on public ECR — no need to build your own:

export NCCL_TEST_IMAGE=public.ecr.aws/hpc-cloud/nccl-tests:cuda12.8.1-efa1.43.2-ofiv1.16.3-ncclv2.27.7-1-testsv2.16.9

Run NCCL Test (MPIJob)

envsubst '${NCCL_TEST_IMAGE}' < infrastructure/manifests/nccl-tests-mpijob.yaml | kubectl apply -f -

# Watch pods
watch kubectl get pods -o wide

# View logs
kubectl logs -f $(kubectl get pods | grep launcher | cut -d ' ' -f 1)

What to Check

1. EFA provider selected: Look for NET/OFI Selected Provider is efa (NOT socket)
2. No crashes: No cm_req_handle_error_entry or Caught signal 11
3. Reasonable bandwidth: g6.8xlarge should show ~2-3 GB/s BusBW at 1GB
4. All GPUs participating: Verify -np matches expected GPU count

Cleanup

kubectl delete mpijob nccl-tests

Only after Phase 0 passes, proceed to purchase capacity blocks for production training.

Level 0: Infrastructure Smoke Tests

GPU Availability

kubectl run gpu-test --rm -it \
  --image=nvcr.io/nvidia/pytorch:24.07-py3 \
  --overrides='{
    "spec": {
      "nodeSelector": {"role": "gpu-training"},
      "tolerations": [{"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}],
      "containers": [{
        "name": "gpu-test",
        "image": "nvcr.io/nvidia/pytorch:24.07-py3",
        "command": ["nvidia-smi"],
        "resources": {"requests": {"nvidia.com/gpu": "8"}, "limits": {"nvidia.com/gpu": "8"}}
      }]
    }
  }'

Expected: 8 GPUs visible with correct type (A100 80GB or H100).

EFA Availability

kubectl exec <gpu-pod> -- ls /sys/class/infiniband/
# Expected: efa0, efa1, ... (4 for p4de, 32 for p5)

Storage Availability

kubectl exec <gpu-pod> -- bash -c "
  echo 'Write test...' && echo 'test' > /fsx/test_write && \
  echo 'Read test...' && cat /fsx/test_write && \
  echo 'Delete test...' && rm /fsx/test_write && \
  echo 'Storage OK'
"

Level 1: Runtime Validation

kubectl exec <gpu-pod> -- python -c "
import torch
print(f'PyTorch: {torch.__version__}')
print(f'CUDA: {torch.version.cuda}')
print(f'cuDNN: {torch.backends.cudnn.version()}')
print(f'GPUs: {torch.cuda.device_count()}')
assert torch.cuda.is_available(), 'CUDA not available!'
assert torch.cuda.device_count() == 8, f'Expected 8 GPUs, got {torch.cuda.device_count()}'

# Test key imports
import verl  # Replace with your RL framework
from flash_attn import flash_attn_func
import transformers
import ray
print('All imports OK')
"

Level 2: Single-GPU Model Test

kubectl exec <gpu-pod> -- python -c "
import torch
from transformers import AutoModelForVision2Seq, AutoTokenizer

model_path = '/fsx/models/openvla-7b'  # Replace with your model path
print(f'Loading model from {model_path}...')

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForVision2Seq.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map='cuda:0'
)

params = sum(p.numel() for p in model.parameters())
print(f'Model loaded: {params / 1e9:.1f}B parameters')
print(f'Tokenizer vocab: {tokenizer.vocab_size}')
assert params > 6e9, f'Model too small: {params}'
print('Single-GPU model test PASSED')
"

Level 3: Multi-GPU FSDP Test

kubectl exec <gpu-pod> -- python -c "
import torch
import torch.distributed as dist
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
import os

os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'

# Init process group for single-node multi-GPU
dist.init_process_group('nccl', rank=0, world_size=1)

# Test NCCL allreduce
tensor = torch.ones(1024, device='cuda:0')
dist.all_reduce(tensor)
print(f'AllReduce result: {tensor[0].item()} (expected 1.0)')

dist.destroy_process_group()
print('Multi-GPU NCCL test PASSED')
"

NCCL Performance Test (Multi-Node via MPIJob)

Discovery #46: The canonical NCCL test on EKS uses an MPIJob (Kubeflow MPI Operator), NOT a StatefulSet with torchrun.

See infrastructure/manifests/nccl-tests-mpijob.yaml. Adjust for your instance type:

Expected results (2x p5.48xlarge):

• 1 GB message: ~437 GB/s BusBW
• 16 GB message: ~487 GB/s BusBW

Level 4: Rollout Test

Test that the simulation environment initializes and the model generates actions:

kubectl exec <gpu-pod> -- python -c "
import os, sys

# CRITICAL: Many robotics simulation libraries call input() at module import
# time if config directories aren't set. This will HANG in non-interactive
# containers. Always set env vars AND redirect stdin before importing.
os.environ['ROBOT_PLATFORM'] = 'LIBERO'  # Replace with your simulation platform
os.environ['LIBERO_DATASET_DIR'] = '/fsx/datasets/libero'  # Replace with your dataset env var and path
sys.stdin = open('/dev/null')  # Prevent input() hang

# Test simulation environment
import libero.libero  # Replace with your simulation library
# ... load benchmark/tasks and verify
print('Rollout environment test PASSED')
"

Example (RLinf): The RLinf reference tests LIBERO with os.environ['ROBOT_PLATFORM'] = 'LIBERO', os.environ['LIBERO_DATASET_DIR'] = '/fsx/datasets/libero', then imports libero.libero and verifies 10 tasks via get_benchmark('libero_10'). See rlinf-functional-testing skill for details.

General pattern: Always test simulation library imports in a non-interactive container (Kubernetes pod, CI) before assuming they work. Many robotics libraries assume interactive terminals.

Level 5: End-to-End Training Step

Run a minimal training step (1 epoch, small batch):

# test-training-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: test-training-e2e
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 3600  # 1 hour timeout
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        role: gpu-training
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: test
          image: <ECR_URI>:latest
          command:
            - bash
            - -c
            - |
              # Run framework-specific pre-training setup
              # Then launch a single training step with minimal data
              bash /workspace/eks/scripts/run_training_eks.sh --max-steps 1  # Replace with your launch script
          resources:
            requests:
              nvidia.com/gpu: 8
              vpc.amazonaws.com/efa: 4
            limits:
              nvidia.com/gpu: 8
              vpc.amazonaws.com/efa: 4
          volumeMounts:
            - name: fsx
              mountPath: /fsx
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: fsx
          persistentVolumeClaim:
            claimName: fsx-claim
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: "64Gi"

Example (RLinf): The RLinf L5 test runs verl.trainer.main_ppo with Hydra overrides for minimal data (n_samples=2, train_batch_size=4, total_epochs=1) and uses overwrite_vla_ckpt_utils.sh for model preparation. The full veRL command includes data.task_suite_name=libero_10, actor_rollout_ref.model.path=/fsx/models/sft-base/openvla-oft-libero10-traj1, and algorithm.adv_estimator=grpo. See rlinf-functional-testing skill for complete configurations.

kubectl apply -f test-training-job.yaml
kubectl logs -f job/test-training-e2e

Expected: Training completes 1 epoch without errors. Validation success rate is reported.

Level 6: Validation-Only Run

Run evaluation to ensure the full inference pipeline works:

# Same as training but with trainer.val_only=True
# This runs greedy rollout across all test scenarios and reports success rates

Automated Test Suite

The repo includes validate.sh, a full validation harness that runs all levels:

# Local validation (no cluster required)
# Runs: terraform fmt/validate, kubeconform, shellcheck, reference integrity
./validate.sh --mode local

# Cluster validation (requires live EKS cluster)
# Runs levels L0-L6 sequentially, stops on first failure
./validate.sh --mode cluster --level 6

# Run up to a specific level
./validate.sh --mode cluster --level 3    # Stop after NCCL/EFA

# Skip earlier levels (resume from a specific level)
./validate.sh --mode cluster --skip-to 2  # Start from container test

# Test a specific example in L5
./validate.sh --mode cluster --level 5 --example maniskill-openvlaoft  # Replace with your example name

# Continue past failures
./validate.sh --mode cluster --continue-on-error

# Override image URIs via environment variables
ECR_URI=123456789.dkr.ecr.us-east-2.amazonaws.com/rlinf-on-eks/rlinf \
  ./validate.sh --mode cluster --skip-to 2 --level 3

Validation Output

========================================
VALIDATION RESULTS
========================================
Level   Name                     Status   Time
-----   ------------------------ ------   ----
L0      Infrastructure           PASS     12s
L1      Container Build          PASS     847s
L2      Container Test           PASS     95s
L3      NCCL/EFA                 PASS     180s
L4      Model Download           PASS     320s
L5      <example-name>           PASS     540s
L5      <example-name>           PASS     510s
L5      <example-name>           PASS     620s
L6      Multi-Node               PASS     480s
========================================
9/9 passed in 3604s

Implementation Details

L5 Training Step Tests

Uses runner.max_steps=1 (or equivalent) override to limit the framework to exactly 1 training step per example. The key insight: find the single override that limits the training loop to one step.

Example (RLinf): RLinf uses runner.max_steps=1 Hydra override to limit to exactly 1 PPO step. RLinf's num_steps_per_epoch is hardcoded to 1.

The training launch script (examples/scripts/<launch-script>) supports HYDRA_OVERRIDES env var for injecting additional overrides without modifying the script:

Common Testing Pitfalls for Physical AI Stacks

These issues affect any RL training deployment, not just specific frameworks:

1. Interactive Prompts in Simulation Libraries

Many robotics simulation libraries (LIBERO, RoboSuite, Habitat, etc.) call input() at module import time to configure dataset directories. In non-interactive containers, this causes the pod to hang indefinitely with no error message.

Fix: Set all required environment variables (e.g., LIBERO_DATASET_DIR) via the pod spec env block AND redirect stdin in the test script: sys.stdin = open('/dev/null').

2. Model/Config Version Skew

Model checkpoints are saved with a config.json that reflects the code version at save time. When the modeling code evolves and adds new config attributes (e.g., use_proprio, num_action_chunks), older checkpoints lack these attributes and from_pretrained() fails with AttributeError.

Fix: Apply patches that use getattr(config, 'attr', default) instead of direct attribute access. Store patches in examples/patches/ for traceability.

3. Container Image Pull Timeouts

Physical AI RL images are 15-20 GB. First pull takes 5-10 minutes. If activeDeadlineSeconds is set too low, test jobs timeout during image pull -- before any code runs.

Fix: Set activeDeadlineSeconds to at least 600s (10 min) for any job using a large image. Use imagePullPolicy: IfNotPresent (not Always) to avoid re-pulling on every run. Tag images with specific versions.

4. YAML/Bash Script Interactions

Complex bash scripts embedded in YAML | block scalars fail in subtle ways. Semicolons, curly braces, and &&/|| chains can be misinterpreted by the YAML parser.

Fix: Keep inline bash simple. For complex test logic, write a Python script and mount it via ConfigMap.

5. GPU Memory Not Freed Between Tests

If running multiple test levels in a single pod, GPU memory from one test may not be freed for the next. Model loading followed by FSDP wrapping can OOM.

Fix: Run each test level in a separate pod, or explicitly del model; torch.cuda.empty_cache(); gc.collect() between levels.

Validation Checklist

• Level 0: All 8 GPUs visible, EFA devices present, FSx mounted
• Level 1: PyTorch + CUDA + all imports succeed
• Level 2: VLA model loads on a single GPU
• Level 3: FSDP wraps model, NCCL allreduce works
• Level 4: Simulation environment initializes
• Level 5: One training epoch completes without errors
• Level 6: Validation run produces success rate metrics

Related Skills

• Skill 09: Kubernetes Manifests - Manifests used in tests
• Skill 12: Deployment Method - Pre-flight checks
• Skill 14: Benchmarking - Performance evaluation