Aks Troubleshooting

AKS Troubleshooting

Root-cause live AKS incidents with a read-only, evidence-first investigation. This skill covers the full Day-2 troubleshooting surface — workloads, nodes, networking, ingress, upgrades, and spot/zone disruptions — and produces a structured incident report.

Operating rules

Read-only by default. Do not restart, delete, cordon, drain, scale, upgrade, or reconfigure any resource unless the user explicitly asks for remediation. Gather evidence, name the root cause, and propose the fix — but do not apply it uninvited.

Evidence before conclusion. Do not state a root cause without quoting the evidence that supports it. "Pod is Pending" and "node is NotReady" are symptoms, not causes — trace them to the specific selector, taint, exhausted resource, or Azure-side condition.

Tool preference. Inspect the host's available tools for Azure MCP capabilities that advertise AKS, AppLens, Azure Monitor, or Resource Health operations. Use each matching capability under its host-assigned name, preferring the smallest operation that fits the read. Never treat a specific prefix or spelling as an availability check, and do not invent a name-mapping layer. Fall back to raw az aks and kubectl only when the host exposes no matching capability or the discovered MCP surface cannot perform the check. Default the MCP access mode to readonly. See references/aks-mcp.md.

Evidence order. Gather Azure-side state first (cluster state, resource health, recent operations, node-pool state, detector/monitoring output), then Kubernetes-side state (reachability, nodes, kube-system, events, the affected namespace, pod detail, logs). This ordering catches platform-level causes — a failed upgrade operation, a stopped cluster, a quota block — before you spend time inside the cluster.

Route by symptom

references/symptom-map.md is the fastest path: 16 symptom sections, each a self-contained block of the exact kubectl/az commands to run plus the common causes. Start there when the symptom is clear; use the topic files above for deeper investigation.

Scripts

Both are POSIX sh, read-only, and safe to run at the start of any investigation. Invoke with sh <script> or ./<script>.

• scripts/cluster-snapshot.sh — quick cluster-health overview (nodes, system pods, recent events, node-pool state).
• scripts/pod-deep-dive.sh <namespace> <pod> — full diagnostic dump for one pod: describe, events, current and previous logs across all containers, resource usage.

AKS-specific gotchas

The highest-signal failure patterns that are specific to AKS — a frontier model will not reliably know these. Review before investigating.

• Azure CNI vs kubenet is a fork in every networking fix. Check az aks show -o json --query networkProfile.networkPlugin first — the plugin (kubenet, Azure CNI, CNI Overlay, Cilium) changes how pod IPs, routes, and network policy behave.
• Managed-identity RBAC is behind a large share of AKS failures. ACR pull, disk attach, private DNS, and Key Vault access all depend on the cluster or kubelet identity having a role assignment. Check az aks show --query identityProfile and the relevant role assignments early.
• Node NotReady is not always a VM problem. It can be kubelet, containerd, the CNI plugin, Azure host maintenance, or an expired kubelet/API-server certificate. Correlate kubectl describe node conditions with az vm get-instance-view, and check kubectl get csr for pending certificate requests.
• The Azure LB health probe can disagree with Kubernetes. A Service can look healthy in-cluster but fail at the Azure load balancer because the LB rule's probe path/port does not match the app endpoint. Check az network lb probe list.
• Subnet exhaustion silently blocks scheduling. Azure CNI allocates a VNet IP per pod; a full pod subnet stops new pods scheduling with no obvious error. Check az network vnet subnet show --query '{addressPrefix: addressPrefix, used: ipConfigurations | length(@)}'.
• System-pool PodDisruptionBudgets block drains during upgrades. CoreDNS and metrics-server ship PDBs that can stall a node drain. Check kubectl get pdb -A.
• The API server IP can change after stop/start. When a cluster is stopped and restarted, the API server IP may change; flush DNS and re-run az aks get-credentials if kubectl cannot connect afterward.
• Private clusters need in-VNet access. kubectl must run from a VM inside — or peered to — the cluster VNet. Check az aks show --query apiServerAccessProfile for private-cluster and authorized-IP-range settings.
• NSG/firewall egress blocks surface as VM extension errors. AKS nodes need outbound access to required FQDNs (AKS API, MCR, management.azure.com, and others). A restrictive NSG or firewall causes VM extension errors during create/upgrade — error codes 50 (OutboundConnFailVMExtensionError), 51 (K8SAPIServerConnFailVMExtensionError), 52 (K8SAPIServerDNSLookupFailVMExtensionError). Check az network nsg rule list and firewall logs.
• SNAT port exhaustion appears past a few hundred nodes. Large clusters using the Azure Load Balancer for outbound can exhaust SNAT ports, causing intermittent egress failures. Check az network lb show --query outboundRules; fix by moving to a NAT gateway (az aks update --outbound-type managedNATGateway).
• Upgrade max-surge defaults to one node at a time. Large-cluster upgrades take hours at the default. Check az aks nodepool show --query upgradeSettings and raise --max-surge if the workload tolerates it.
• kubectl must be within two minor versions of the cluster. A stale client produces confusing errors. Compare kubectl version --client with az aks show --query kubernetesVersion.

Log discipline

• Always fetch kubectl logs --previous alongside current logs — after a restart the current stream may be empty.
• Do not truncate logs with --tail or | tail; the causal error is often early.
• For multi-container pods, use --all-containers (or name each --container) so sidecar and init-container logs are not missed.
• Get current UTC time with date -u before using --since-time.

Deep diagnostics

When standard checks do not reveal a root cause, use Inspektor Gadget for real-time, low-level node and pod observability (DNS traces, TCP traces, process and file-access snapshots). See references/inspektor-gadget.md for the gadget catalog and symptom-to-gadget mapping. Inspektor Gadget runs a privileged debug pod — get explicit user approval before invoking it. Additional MCP-driven investigation modes are in references/structured-input-modes.md and references/command-flows.md.

Report

Structure the final incident report using references/report-template.md: symptom and impact, evidence gathered, failure domain, root cause with supporting evidence, confidence, remediation, and escalation. Quote relevant log snippets inline rather than pasting full dumps.

Reference

Microsoft's AKS troubleshooting hub: https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/welcome-azure-kubernetes