Monitoring Observability

Skill 15: Monitoring and Observability

Purpose

Monitor GPU utilization, training progress, infrastructure health, and costs for RL training workloads running on EKS. This covers metrics collection, dashboards, alerting, and cost tracking.

Monitoring Stack

+-------------------------------------------------------------------+
|                       Monitoring Architecture                      |
|                                                                    |
|  GPU Nodes                                                         |
|  ┌─────────────────┐                                              |
|  │ DCGM Exporter   │──► Prometheus ──► Grafana Dashboards         |
|  │ (GPU metrics)   │                                              |
|  ├─────────────────┤                                              |
|  │ Node Exporter   │──► Prometheus ──► Grafana Dashboards         |
|  │ (CPU/mem/disk)  │                                              |
|  ├─────────────────┤                                              |
|  │ Training Pod    │──► MLflow (training metrics)                  |
|  │ (RL framework)  │──► Ray Dashboard (cluster state)             |
|  └─────────────────┘                                              |
|                                                                    |
|  CloudWatch                                                        |
|  ┌─────────────────┐                                              |
|  │ EKS metrics     │──► CloudWatch Dashboards + Alarms            |
|  │ FSx metrics     │                                              |
|  │ EC2 metrics     │                                              |
|  └─────────────────┘                                              |
+-------------------------------------------------------------------+

Layer 1: GPU Metrics (DCGM Exporter)

Install DCGM Exporter

DCGM Exporter is managed by the NVIDIA GPU Operator (see Skill 01 helm.tf). The GPU Operator Helm release sets dcgmExporter.enabled = true, which deploys DCGM exporter pods on all GPU nodes automatically.

Discovery #59: The NVIDIA GPU Operator creates the DCGM exporter DaemonSet and Service (nvidia-dcgm-exporter on port 9400) but does NOT create a ServiceMonitor. Without a ServiceMonitor, Prometheus never discovers or scrapes GPU metrics. Our infrastructure/addons/main.tf creates the ServiceMonitor via null_resource.dcgm_service_monitor (using kubectl apply because kubernetes_manifest requires CRD discovery at plan time, which fails on fresh clusters). The ServiceMonitor targets the gpu-operator namespace with label selector app: nvidia-dcgm-exporter and scrapes the gpu-metrics port at 15s intervals.

For manual standalone installation (not needed when using GPU Operator):

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install dcgm-exporter nvidia/dcgm-exporter \
  --namespace monitoring \
  --create-namespace \
  --set tolerations[0].key=nvidia.com/gpu \
  --set tolerations[0].operator=Exists \
  --set tolerations[0].effect=NoSchedule \
  --set nodeSelector.role=gpu-training

Key GPU Metrics

Layer 2: Cluster Metrics (Prometheus + Grafana)

Install kube-prometheus-stack

The kube-prometheus-stack is installed as a Helm release in Terraform (see Skill 01 helm.tf). For manual installation:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=your-password \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false

Access Grafana

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
# Open http://localhost:3000

Custom Dashboard: RL Training Overview

Discovery #60: Grafana sidecar dashboard provisioning works via ConfigMaps. The kube-prometheus-stack deploys a Grafana sidecar container that watches for ConfigMaps with the label grafana_dashboard: "1". Any ConfigMap with this label in the monitoring namespace is automatically imported as a Grafana dashboard. Create a kubernetes_config_map resource in infrastructure/addons/main.tf with the dashboard JSON and the grafana_dashboard: "1" label.

Create a Grafana dashboard with these panels:

GPU Section:

# Average GPU utilization across training pods
avg(DCGM_FI_DEV_GPU_UTIL{pod=~"training.*"})

# GPU memory usage per GPU
DCGM_FI_DEV_FB_USED{pod=~"training.*"}

# GPU temperature
DCGM_FI_DEV_GPU_TEMP{pod=~"training.*"}

Node Section:

# CPU utilization (for env rendering)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes

# Network throughput (EFA)
rate(node_network_transmit_bytes_total{device=~"eth.*"}[5m])

Storage Section:

# FSx IOPS (from CloudWatch via YACE exporter)
# Or check via: kubectl exec <pod> -- lctl get_param osc.*.stats

Layer 3: Training Metrics (MLflow)

The RL framework logs training metrics to MLflow:

Key Metrics to Monitor

MLflow Configuration (Optional)

MLflow is an optional addon (disabled by default). Enable it with enable_mlflow = true in infrastructure/addons/terraform.tfvars. When enabled, configure the tracking URI in your training pod:

# In training pod environment (only when MLflow addon is enabled)
export MLFLOW_TRACKING_URI="http://mlflow.rlinf.svc.cluster.local"

# Framework-specific logger configuration
# (exact syntax depends on the RL framework)

Example (RLinf): The RLinf reference uses Hydra overrides: trainer.logger=['console','mlflow'], trainer.experiment_name=RLinf. See examples/AGENTS.md for details.

MLflow Alerts

MLflow does not include a built-in alerting system. Use Grafana alerting rules against Prometheus metrics for equivalent coverage:

• Training stall: Alert when DCGM_FI_DEV_GPU_UTIL drops to 0 for >30 minutes
• NaN loss: Configure a Grafana alert on the actor/loss metric panel
• Success rate drop: Alert on sudden decrease in validation success rate via Grafana
• GPU OOM: Alert when pod restarts detected via kube_pod_container_status_restarts_total

Layer 4: CloudWatch (AWS Infrastructure)

EKS Container Insights

# Enable Container Insights
aws eks create-addon \
  --cluster-name <cluster-name> \
  --addon-name amazon-cloudwatch-observability \
  --resolve-conflicts OVERWRITE

FSx for Lustre Metrics

Monitor in CloudWatch:

• DataReadBytes / DataWriteBytes -- I/O throughput
• FreeDataStorageCapacity -- Available space
• MetadataOperations -- Metadata IOPS

CloudWatch Alarms

# Alarm: FSx running low on space
aws cloudwatch put-metric-alarm \
  --alarm-name fsx-low-space \
  --metric-name FreeDataStorageCapacity \
  --namespace AWS/FSx \
  --statistic Average \
  --period 300 \
  --threshold 500000000000 \
  --comparison-operator LessThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:<ACCOUNT>:alerts

# Alarm: GPU node not reporting
aws cloudwatch put-metric-alarm \
  --alarm-name gpu-node-missing \
  --metric-name node_count \
  --namespace ContainerInsights \
  --dimensions Name=ClusterName,Value=<cluster-name> \
  --statistic Minimum \
  --period 300 \
  --threshold 1 \
  --comparison-operator LessThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:us-east-1:<ACCOUNT>:alerts

Layer 5: Ray Dashboard

Ray provides a built-in dashboard for cluster monitoring:

# Port-forward Ray dashboard
kubectl port-forward <training-pod-0> 8265:8265
# Open http://localhost:8265

The Ray dashboard shows:

• Worker status (alive, dead, pending)
• Resource utilization (CPU, GPU, memory per worker)
• Task execution timeline
• Object store usage

Cost Monitoring

AWS Cost Explorer Tags

Tag all resources for cost tracking:

tags:
  Project: <cluster-name>
  Environment: training
  Experiment: <experiment-name>

Cost Estimation

# Quick cost estimate for a training run
# p4de.24xlarge: ~$41/hr per node
# 2 nodes x 100 hours = ~$8,200
# FSx 4.8 TiB PERSISTENT_2 at 125 MB/s: ~$696/month
# S3 storage: ~$23/TB/month

Cost Optimization Alerts

# Budget alert: warn when monthly spend exceeds threshold
aws budgets create-budget \
  --account-id <ACCOUNT_ID> \
  --budget '{
    "BudgetName": "<cluster-name>-monthly",
    "BudgetLimit": {"Amount": "10000", "Unit": "USD"},
    "TimeUnit": "MONTHLY",
    "BudgetType": "COST",
    "CostFilters": {"TagKeyValue": ["user:Project$<cluster-name>"]}
  }' \
  --notifications-with-subscribers '[{
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 80,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{"SubscriptionType": "EMAIL", "Address": "team@example.com"}]
  }]'

Troubleshooting with Observability

Validation Checklist

• DCGM Exporter running on all GPU nodes
• Prometheus scraping GPU and node metrics
• Grafana dashboard shows GPU utilization, memory, temperature
• MLflow receiving training metrics (loss, success rate, timing)
• CloudWatch Container Insights enabled
• CloudWatch alarms set for FSx space and node availability
• Cost tracking tags applied to all resources
• Budget alerts configured

Related Skills

• Skill 10: Kubernetes Native Features - NVIDIA GPU Operator
• Skill 14: Benchmarking - Performance baseline metrics
• Skill 01: EKS Cluster Provisioning - Cluster-level monitoring setup