Storage S3 Files

Skill 05d: Storage - Amazon S3 Files

Purpose

Use Amazon S3 Files to mount S3 buckets as native NFS file systems on EKS. S3 Files (launched April 2026) is a shared file system built on Amazon EFS that provides low-latency file access to data stored in S3. It is the first cloud object store to offer fully-featured NFS v4.1+ file system access directly to bucket data.

S3 Files is suitable for:

• Development and debugging workloads that need file-system access to S3 data
• Dataset browsing and preparation before training
• Inference serving (read-heavy, large sequential reads streamed from S3)
• Cost-sensitive persistent environments where FSx for Lustre's minimum 1200 GiB is overkill
• Multi-compute shared workspaces (up to 25,000 concurrent connections)

When to Use S3 Files vs Other Storage Options

Why FSx for Lustre Remains Recommended for Training

For active Physical AI RL training (RL framework with GRPO), FSx for Lustre is the better choice because:

1. POSIX consistency: RL framework workers on multiple nodes write checkpoints and shared state. FSx provides full POSIX consistency; S3 Files provides NFS close-to-open consistency (writes not visible to other clients until file is closed and reopened). This can cause subtle issues with frameworks that expect immediate cross-node visibility.

2. Checkpoint I/O patterns: Training checkpoints involve rapid overwrites and renames. FSx handles these atomically. S3 Files translates renames into copy+delete operations on S3 (non-atomic, with synchronization delay).

3. Predictable latency: FSx provides sub-ms latency for all operations. S3 Files has first-access latency (metadata import on first directory listing) that can add seconds to job startup.

4. Cost context: FSx at $174/month for 1200 GiB is negligible compared to GPU costs ($2/hr per g6, $36/hr per p5). Optimizing storage cost is not the priority during active training.

Architecture

EKS Training Pod              S3 File System                Amazon S3
+------------------+         +--------------------+        +------------------+
| /s3files/        |  NFS    | High-perf storage  |        | s3://bucket/     |
| +-- datasets/    | <-----> | (EFS-backed, ~1ms) | <----> | +-- datasets/    |
| +-- models/      |  v4.1+  |                    |  sync  | +-- models/      |
| +-- outputs/     |         | Small files cached  |        | +-- outputs/     |
+------------------+         | Large reads -> S3   |        +------------------+
                             +--------------------+
                             Mount target in VPC
                             (one per AZ)

Data Flow

• Small file reads (<128KB): Served from high-performance storage at ~1ms latency
• Large file reads (>=128KB): Streamed directly from S3 at TB/s throughput (no S3 Files storage charge)
• Writes: Go to high-performance storage (durable immediately), synced to S3 within ~60s
• Unused data: Automatically expired from high-perf storage after configurable window (1-365 days, default 30)

Performance Specifications

Pricing

S3 Files has no provisioned capacity. You pay for:

1. High-performance storage: $0.30/GB-month (only for actively cached data)
2. Data writes (to high-perf storage): $0.06/GB
3. Data reads (from high-perf storage): $0.03/GB
4. Large reads (>=128KB): Streamed from S3 at S3 GET request rates -- no S3 Files charge
5. S3 storage: Standard S3 rates still apply for data in the bucket

Cost Example for Training on EKS

If active working set is ~50GB (datasets loaded, a few checkpoints cached):

• High-perf storage: 50GB x $0.30 = $15/month
• Plus S3 storage and access charges

Compare with FSx for Lustre:

• 1200 GiB PERSISTENT_2 @ $0.145/GB = $174/month (fixed, regardless of usage)

Step-by-Step

Prerequisites

• An S3 general-purpose bucket (e.g., <bucket-name>)
• S3 Versioning enabled on the bucket (required by S3 Files)
• EKS cluster in the same VPC
• EFS CSI driver installed (S3 Files uses the same CSI driver as EFS)

1. Create the S3 File System

Via AWS CLI

export AWS_REGION=us-east-2

# Create the file system linked to your S3 bucket
aws s3api create-file-system \
  --bucket <bucket-name> \
  --file-system-configuration '{
    "PerformanceMode": "generalPurpose",
    "ThroughputMode": "elastic"
  }' \
  --region $AWS_REGION

# Note the FileSystemId from the output (e.g., fs-0aa860d05df9afdfe)

Via Terraform (future integration)

At the time of writing (April 2026), S3 Files is new and Terraform provider support may be limited. Check the aws provider changelog for aws_s3_file_system resource availability. Until then, use CLI or console to create the file system and manage the mount targets.

2. Create Mount Targets

Mount targets provide network access to the file system within your VPC. You need one per AZ where your pods run.

# Get the VPC subnet for your target AZ
SUBNET_ID="subnet-0c0eba4f645d647cc"  # us-east-2a private subnet

# Get the node security group (pods need NFS port 2049 access)
NODE_SG=$(aws eks describe-cluster --name <cluster-name> \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text \
  --region $AWS_REGION)

# Create a security group for the mount target
MT_SG=$(aws ec2 create-security-group \
  --group-name <cluster-name>-s3files-mt \
  --description "S3 Files mount target SG" \
  --vpc-id vpc-0d0cced196d7bda32 \
  --query "GroupId" --output text \
  --region $AWS_REGION)

# Allow NFS from EKS nodes
aws ec2 authorize-security-group-ingress \
  --group-id $MT_SG \
  --protocol tcp \
  --port 2049 \
  --source-group $NODE_SG \
  --region $AWS_REGION

# Create the mount target
aws s3api create-mount-target \
  --file-system-id fs-0aa860d05df9afdfe \
  --subnet-id $SUBNET_ID \
  --security-groups $MT_SG \
  --region $AWS_REGION

3. Install the EFS CSI Driver (if not already installed)

S3 Files uses the EFS CSI driver (efs.csi.aws.com). If you already have Amazon EFS configured, the same driver works for S3 Files.

Via EKS Managed Add-on

# Create IAM role with AmazonS3FilesCSIDriverPolicy
# (See AWS docs for the managed policy ARN)

aws eks create-addon \
  --cluster-name <cluster-name> \
  --addon-name aws-efs-csi-driver \
  --service-account-role-arn arn:aws:iam::<account-id>:role/<cluster-name>-efs-csi \
  --region $AWS_REGION

Via Terraform

# In main.tf cluster_addons block
aws-efs-csi-driver = {
  most_recent = true
  pod_identity_association {
    role_arn        = aws_iam_role.efs_csi.arn
    service_account = "efs-csi-controller-sa"
  }
}

4. Create PersistentVolume and PersistentVolumeClaim

# s3files-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3files-pv
spec:
  capacity:
    storage: 1Ti  # Informational -- S3 Files scales automatically
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: efs.csi.aws.com
    # volumeHandle format: <file-system-id>::<access-point-id>
    # Use the access point created by S3 Files
    volumeHandle: fs-0aa860d05df9afdfe::fsap-0123456789abcdef0
---
# s3files-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3files-claim
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 1Ti
  volumeName: s3files-pv

kubectl apply -f s3files-pv.yaml
kubectl get pvc s3files-claim  # Should show Bound

5. Mount in Pods

spec:
  containers:
    - name: workload
      image: <your-image>
      volumeMounts:
        - name: s3files
          mountPath: /s3data
  volumes:
    - name: s3files
      persistentVolumeClaim:
        claimName: s3files-claim

6. Verify

# Test pod
kubectl run s3files-test --image=amazonlinux:2023 --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "test",
        "image": "amazonlinux:2023",
        "command": ["sh", "-c", "ls -la /s3data && echo hello > /s3data/test.txt && cat /s3data/test.txt && sleep 120"],
        "volumeMounts": [{"name": "s3files", "mountPath": "/s3data"}]
      }],
      "volumes": [{"name": "s3files", "persistentVolumeClaim": {"claimName": "s3files-claim"}}]
    }
  }'

# Check output
kubectl logs s3files-test

# Verify file appeared in S3 (after ~60s sync delay)
aws s3 ls s3://<bucket-name>/test.txt --region us-east-2

# Cleanup
kubectl delete pod s3files-test

Synchronization Behavior

File System to S3

• Writes are durable immediately on high-performance storage
• Changes are batched for ~60 seconds before syncing to S3 (reduces PUT request costs)
• Export rate: up to 800 files/s, 2,700 MB/s

S3 to File System

• Changes in S3 appear in file system within seconds (via S3 Event Notifications)
• Only files with data on high-perf storage are updated immediately
• Expired files are updated on next access
• Import rate: up to 2,400 objects/s, 700 MB/s

Conflict Resolution

• S3 is the source of truth: If the same file is modified on both the file system and S3 before sync completes, the S3 version wins
• Conflicting file system version is moved to a .nfs-conflicts/ directory

First Access Latency

• First ls of a directory imports metadata for all files in that directory
• Files below the import size threshold (default 128KB) have their data imported asynchronously
• Subsequent directory listings and file access are fast

Limitations

Training Framework Integration

Reference-specific details: See examples/AGENTS.md for framework-specific S3 Files integration patterns and recommended use cases.

Recommended Use Cases (Not Training)

S3 Files is well-suited for non-training workflows in the RL pipeline:

Not Recommended For

• Active RL training: Use FSx for Lustre (POSIX consistency, predictable latency)
• FSDP checkpoint saves: Rapid overwrites and gathers across ranks need POSIX semantics
• MLflow logging: Updates files in-place; sync delay may lose recent logs on crash

Validation Checklist

• S3 file system is in AVAILABLE state
• Mount target exists in the target AZ
• Security group allows NFS port 2049 from EKS nodes
• EFS CSI driver pods are running
• PV and PVC are bound
• Test pod can ls mounted path and see S3 objects as files
• Test pod can write a file and verify it appears in S3 (~60s delay)
• Multi-node test: pods on different nodes see the same data

Related Skills

• Skill 05: Storage - FSx for Lustre - Recommended for active training (POSIX, sub-ms latency)
• Skill 05b: Storage - S3 Connector for PyTorch - Programmatic S3 access from training code
• Skill 05c: Storage - Mountpoint for S3 - FUSE-based S3 mount (read-heavy, no NFS overhead)
• Skill 06: Dataset Preparation - Staging data on S3