AKS Memory Saturation Troubleshooting

AKS Memory Saturation Troubleshooting

Overview

Memory saturation occurs when applications or processes need more memory than the container host can provide, or when the host exhausts available memory.

Identification Methods

Method 1: Container Insights (Azure Portal)

1. Navigate to Kubernetes services → Select cluster
2. Go to Monitoring → Insights
3. Select Nodes tab
4. Choose metric: Memory working set (computed from Allocatable)
5. Set percentiles to Max
6. Sort by Max % column

Method 2: kubectl top

# Node memory usage
kubectl top node

# Example output
NAME                                STATUS   CPU(cores)   MEMORY(bytes)   MEMORY%
aks-agentpool-12345678-vmss000000   Ready    250m         4500Mi          69%
aks-agentpool-12345678-vmss000001   Ready    180m         3000Mi          46%

Method 3: Inspektor Gadget (Advanced)

# Top 10 memory-consuming processes cluster-wide
kubectl gadget run top_process --sort -memoryRelative --max-entries 10

# By specific node
kubectl gadget run top_process --sort -memoryRelative --filter k8s.node==<node-name>

# By namespace
kubectl gadget run top_process --sort -memoryRelative --filter k8s.namespace==<namespace>

# By pod
kubectl gadget run top_process --sort -memoryRelative --filter k8s.podName==<pod-name>

Symptoms and Indicators

Symptom Progression

Memory Pressure → Pod Eviction → Node Not Ready → OOM Kill

Diagnostic Commands

Basic Diagnostics

# Node memory usage
kubectl top node

# Describe node (shows allocatable resources)
kubectl describe node <node-name>

# Node conditions
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="MemoryPressure")].status}{"\n"}{end}'

Detailed Pod Memory Analysis

# Pod memory usage on specific node
kubectl get pods --all-namespaces --output wide \
    | grep <node-name> \
    | awk '{print $1" "$2}' \
    | xargs -n2 kubectl top pods --namespace \
    | awk 'NR==1 || NR%2==0' \
    | sort -k3n \
    | column -t

Memory Events

# OOMKilled events
kubectl get events --all-namespaces --field-selector reason=OOMKilling

# Memory pressure events
kubectl get events --all-namespaces | grep -i memory

Resolution Steps

Step 1: Implement Memory Requests and Limits

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "180Mi"      # Close to actual usage
      limits:
        memory: "300Mi"      # Prevent overcommitting

Guidelines:

• Set requests close to actual usage
• Set limits slightly higher as buffer
• Monitor actual usage to tune values

Step 2: Enable Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

Step 3: Apply Pod Anti-Affinity

apiVersion: apps/v1
kind: Deployment
metadata:
  name: high-memory-app
spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - high-memory-app
            topologyKey: kubernetes.io/hostname

Step 4: Scale Up VM SKUs

# Create new node pool with higher memory VMs
az aks nodepool add \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name highmempool \
  --node-count 3 \
  --node-vm-size Standard_E8s_v3

# Cordon existing nodes
kubectl cordon <old-node>

# Drain workloads
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data

# Delete old node pool
az aks nodepool delete \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name oldpool

Step 5: Separate Workload Types

# Dedicate node pool for applications
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      nodeSelector:
        agentpool: userpool
      tolerations:
      - key: "workload"
        operator: "Equal"
        value: "user"
        effect: "NoSchedule"

Prevention Strategies

Resource Quota for Namespaces

apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-quota
  namespace: production
spec:
  hard:
    requests.memory: "10Gi"
    limits.memory: "20Gi"

Limit Range for Defaults

apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: production
spec:
  limits:
  - default:
      memory: "512Mi"
    defaultRequest:
      memory: "256Mi"
    type: Container

Monitoring Queries

Log Analytics - OOMKilled Containers

KubeEvents
| where Reason == "OOMKilling"
| project TimeGenerated, Namespace, Name, Message
| order by TimeGenerated desc

Memory Working Set by Node

InsightsMetrics
| where Name == "node_memory_working_set_percentage"
| summarize avg(Val) by Computer, bin(TimeGenerated, 5m)
| render timechart

Memory Saturation Alert Query

InsightsMetrics
| where Name == "node_memory_working_set_percentage"
| summarize MaxMemory = max(Val) by Computer
| where MaxMemory > 90

Critical Metrics to Monitor

Best Practices Summary

1. Set appropriate requests/limits for all containers
2. Monitor memory working set against allocatable (not total)
3. Use HPA to scale horizontally before vertical
4. Implement resource quotas per namespace
5. Use node pools to separate workload types
6. Set up alerts for memory pressure conditions
7. Regular capacity planning based on actual usage