Gpu Instance Selection

Skill 02: GPU Instance Selection

Purpose

Select the optimal EC2 GPU instance type for Physical AI RL training workloads on EKS. The choice depends on model size, VRAM requirements, inter-node communication needs, budget, and availability.

Decision Framework

Key Requirements for Physical AI RL

Instance Comparison Matrix

Validation-tier instances (infrastructure testing, NOT for VLA training):

CRITICAL: g6.8xlarge (1x L4 24GB) CANNOT run VLA RL training for 7.5B+ models. FSDP on a single GPU cannot shard -- the full model must fit in VRAM during initialization. Even with all offloading enabled (param, grad, optimizer offload + gradient checkpointing), the L4 OOMs during flatten_tensors_into_flat_param. Use g6 for infrastructure validation (EFA, NCCL, container tests) only.

Note: Pricing is approximate and changes. Always verify current pricing at https://aws.amazon.com/ec2/pricing/on-demand/

Decision Tree

Is VRAM > 40GB per GPU required?
├── YES (7B+ VLA models with FSDP, parallel env rendering)
│   ├── Is maximum training throughput critical?
│   │   ├── YES → p5.48xlarge (H100) or p5e.48xlarge (H200)
│   │   │         - 3200 Gbps EFA for fast allreduce
│   │   │         - NVSwitch 900 GB/s intra-node
│   │   │         - 192 vCPUs for parallel env rendering
│   │   └── NO → p4de.24xlarge (A100 80GB)
│   │             - Best cost/performance for 80GB VRAM
│   │             - 400 Gbps EFA (sufficient for 2-node training)
│   │             - Closest to A800 used in the original paper
│   └── Is budget the primary constraint?
│       └── YES → p4de.24xlarge (A100 80GB)
│                 ~60% cheaper than p5.48xlarge
└── NO (smaller models, experimenting)
    └── p4d.24xlarge (A100 40GB)
        - Lowest cost with EFA
        - May require aggressive memory optimization

VRAM Budget Estimation

For a 7B VLA model with FSDP:

Result: 40GB GPUs are marginal. 80GB GPUs provide comfortable headroom for batch size scaling and parallel environments.

7B+ VLA Model GPU Sizing

Recommended Configuration

RLinf Example: See examples/AGENTS.md for RLinf-specific GPU sizing requirements.

Why p4de.24xlarge is the Default Recommendation

1. Hardware match: Many Physical AI RL frameworks were developed on NVIDIA A800 80GB, which is architecturally identical to A100 80GB (A800 is the China-export variant)
2. Cost efficiency: ~60% cheaper than p5.48xlarge with the same 80GB VRAM
3. EFA bandwidth: 400 Gbps is sufficient for 2-node FSDP training of 7B models
4. Availability: Generally easier to obtain capacity than p5 instances

When to Choose p5.48xlarge

• Training on 4+ nodes where EFA bandwidth becomes the bottleneck (3200 Gbps vs 400 Gbps)
• Faster time-to-result justifies the 2.4x cost premium
• Need higher CPU core count (192 vs 96) for parallel environment rendering
• Need more system RAM (2048 GB vs 1152 GB)

Capacity Planning

Request Quotas Early

GPU instances have strict service quotas. Request increases before you need them:

# Check current quota for p4de instances
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-417A185B \
  --region us-east-1

# Request quota increase (example: 96 vCPUs = 1x p4de.24xlarge)
aws service-quotas request-service-quota-increase \
  --service-code ec2 \
  --quota-code L-417A185B \
  --desired-value 192 \
  --region us-east-1

Common quota codes:

• L-417A185B: Running On-Demand P instances (p4d, p4de)
• L-5480EFD2: Running On-Demand P5 instances

Capacity Reservations

For predictable workloads, use Capacity Reservations (On-Demand Capacity Reservations or Capacity Blocks for ML) to reserve instance capacity in advance:

aws ec2 create-capacity-reservation \
  --instance-type p4de.24xlarge \
  --instance-platform Linux/UNIX \
  --availability-zone us-east-1a \
  --instance-count 2 \
  --instance-match-criteria targeted

Spot vs On-Demand

Related Skills

• Skill 01: EKS Cluster Provisioning
• Skill 03: AMI and Node Configuration