Eks Best Practices

EKS Best Practices

Comprehensive guidance for designing, deploying, and operating Amazon EKS clusters. Consolidates guidance from the AWS EKS Best Practices Guide, AWS EKS HA/Resiliency Guide, and terraform-aws-modules/terraform-aws-eks examples.

When to Use This Skill

Activate this skill when:

• Designing a new EKS cluster architecture
• Choosing between EKS compute options (Fargate, MNG, Karpenter, Auto Mode)
• Configuring EKS networking (VPC CNI, ingress, service mesh)
• Implementing EKS security (IAM, pod security, secrets)
• Planning cluster upgrades or migrations
• Reviewing EKS architecture decisions
• Working with terraform-aws-modules/terraform-aws-eks examples
• Optimizing EKS cost or scaling to large clusters

Don't use this skill for:

• Generic Kubernetes concepts (Claude knows these)
• Provider-specific API reference (link to AWS docs)
• Non-EKS container orchestration (ECS, Lambda)
• Step-by-step EKS upgrade execution — this skill covers upgrade strategy and architectural decisions, not the per-version procedures themselves.

EKS Architecture Decision Framework

When to Use EKS

EKS Deployment Models

For choosing among and operating the on-premises/hybrid models, see: Hybrid & On-Premises Deployments.

Routing: the advisory question "which on-prem/hybrid model should I choose?" stays in this skill; generating a design document or diagram → eks-design; generating a full production cluster Terraform project or an air-gapped build → eks-build (the hybrid cloud/cluster-side Terraform example itself lives in this skill's terraform-examples.md).

Shared Responsibility

Compute Selection Matrix

Decision Table

Quick Decision Guide

• Default choice: Karpenter — best balance of flexibility, cost, and automation
• Zero ops priority: EKS Auto Mode — AWS manages nodes, add-ons, and scaling via managed Karpenter. Best for teams that want Kubernetes benefits without operational overhead around upgrades, autoscaling, load balancing, and storage
• Serverless/batch: Fargate — no nodes to manage, per-pod billing
• Predictable, stable: MNG — familiar ASG model, managed updates
• Custom requirements: Self-managed — full control, highest overhead

✅ DO:

• Use Karpenter as the default node autoscaler for new clusters
• Run system components (CoreDNS, Karpenter) on MNG or Fargate
• Use multiple instance types for availability and cost optimization

❌ DON'T:

• Use self-managed nodes without a specific technical requirement
• Run Fargate for GPU or DaemonSet-dependent workloads
• Mix Karpenter and Cluster Autoscaler on the same node groups

Hybrid & On-Premises Deployments

Three models run EKS outside a standard in-Region cluster. Pick by who owns the control plane and whether the site stays connected:

Critical rule: split by why the site is disconnected. Permanently air-gapped/isolated → EKS Anywhere (the only fully air-gapped model). A connected site that must keep operating through Region outages on AWS-owned hardware (data residency/sovereignty) → Outposts local clusters (Outposts racks only) — disconnect-tolerant, not air-gapped (IAM/IRSA/KMS/EBS-PV/Route 53 are unavailable offline). EKS Hybrid Nodes and Outposts extended clusters both depend on a reliable connection to an AWS Region.

For detailed hybrid-deployment guidance, see: Hybrid & On-Premises Deployments

Networking Quick Reference

VPC CNI Mode Decision

Ingress Pattern Selection

IPv4 vs IPv6

For detailed networking guidance, see: Networking — VPC CNI & IP | Networking — Ingress & DNS

Security Essentials

IAM Strategy

Key rules:

• ✅ Use Pod Identity for new workloads — simpler setup, session tags, role chaining
• ✅ Use EKS access entries (API mode) over aws-auth ConfigMap
• ✅ Move VPC CNI permissions from node role to Pod Identity/IRSA
• ❌ Don't use wildcard conditions in IRSA trust policies
• ❌ Don't attach application permissions to node IAM roles

Pod Security Baseline

Apply Pod Security Admission (PSA) labels to all namespaces:

# Minimum: enforce baseline, warn on restricted
metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted

Secrets Management

Envelope encryption is on by default (≥1.28) and covers all Kubernetes API data; layer your own KMS CMK when you need key control and CloudTrail visibility. Source: https://docs.aws.amazon.com/eks/latest/userguide/envelope-encryption.html

For detailed security guidance, see: Security Reference | Runtime & Network | Supply Chain & Compliance

Reliability Essentials

Pod Disruption Budgets

Create PDBs for every production workload with >1 replica:

Health Probe Strategy

Critical rule: Liveness probes must NOT check external dependencies. If the database goes down and liveness checks the DB, ALL pods restart — causing cascading failure.

Graceful Shutdown Pattern

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 15"]

Why sleep 15: Gives kube-proxy and load balancer time to remove the pod from traffic routing before SIGTERM.

Multi-AZ Distribution

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule

For detailed reliability guidance, see: Reliability & Resiliency — Core (see also reliability-advanced.md for DR, deployment strategies, and large-cluster guidance)

Cluster Upgrade Strategy

Upgrade Sequence (Strict Order)

1. Control Plane → 2. EKS Add-ons → 3. Data Plane → 4. Custom Add-ons

Pre-Upgrade Checklist

1. Check EKS Cluster Insights for upgrade readiness
2. Scan for deprecated APIs (Pluto, kube-no-trouble)
3. Verify add-on compatibility with target version
4. Test in non-prod environment first
5. Ensure PDBs are configured for graceful node drain
6. Back up cluster state (Velero or GitOps repo)

Upgrade Strategy Decision

Native in-place rollback to N-1 is supported within 7 days of an in-place upgrade (all regions; Auto Mode rolls back nodes automatically; rollback into an extended-support version requires setting the cluster upgrade policy to EXTENDED first). update-cluster-version supports type VersionRollback, gated by a ROLLBACK_READINESS insight; add-ons/data-plane roll back separately. Blue-green retains independent rationale for post-7-day windows, multi-minor jumps, and data-plane isolation — not "because you can't roll back."

Data Plane with Karpenter

Karpenter automatically replaces nodes via drift detection after control plane upgrade. Control the speed with disruption.budgets:

disruption:
  budgets:
  - nodes: "10%"  # Max 10% of nodes replaced at a time

For detailed upgrade guidance, see: Cluster Upgrades Reference

Autoscaling Quick Reference

Node Autoscaler Selection

Cluster Autoscaler's ~60-90s scale-up assumes cold EC2 launches; MNG EC2 warm pools (2026-04) pre-initialize instances and cut that latency substantially for MNG-backed node groups.

Pod Autoscaler Selection

For detailed autoscaling guidance, see: Autoscaling Reference | Karpenter Reference

Preparing for a Known Traffic Peak

For a planned peak — a flash sale, marketing push, product launch, or seasonal event — the arrival is a near-instantaneous step at a known clock time, so favor scheduled pre-scaling (set floors ahead of the trigger) over relying on reactive autoscaling to catch up. Pre-warm all three layers (control plane, nodes, pods) and hold the floor across the whole event window rather than scaling down between peaks.

Critical rule: A load test that passes on request rate can still miss the failure that hits production — if it never reproduces the resource-consumption pattern (memory working-set, connection count) of the event's actual access patterns, especially for new features. Test consumption shapes, not just throughput, and act on findings before the event.

For detailed surge-readiness guidance, see: Surge Readiness Reference

Terraform Examples Quick Start

Based on terraform-aws-modules/terraform-aws-eks.

Example Selection

Common Deployment Topologies

Private cluster with Karpenter:

VPC (3 AZs, terraform-aws-modules/vpc/aws)
├── Private subnets → EKS nodes (MNG for system, Karpenter for workloads)
├── Public subnets  → ALB (internet-facing)
├── Intra subnets   → EKS control plane ENIs
└── NAT Gateway     → 1 per AZ for production

Multi-tenant platform:

EKS Cluster (terraform-aws-modules/eks/aws)
├── kube-system         (platform: CoreDNS, kube-proxy, VPC CNI)
├── karpenter           (Karpenter controller on MNG)
├── monitoring          (shared: Prometheus, Grafana)
├── ingress             (shared: AWS LBC)
├── team-a namespace    (RBAC, NetworkPolicy, ResourceQuota)
├── team-b namespace    (RBAC, NetworkPolicy, ResourceQuota)
└── team-c namespace    (RBAC, NetworkPolicy, ResourceQuota)

For detailed examples and terraform patterns, see: Terraform Examples Reference

Cost Optimization Quick Wins

For detailed cost guidance, see: Cost Optimization Reference | For scalability guidance, see: Scalability Reference

Observability Quick Reference

Essential: Enable EKS audit logging and GuardDuty EKS Runtime Monitoring for security visibility.

For detailed observability guidance, see: Observability Reference

EKS Capabilities

EKS Capabilities are AWS-managed features installed and updated as part of the EKS platform. They run in AWS-owned infrastructure separate from your clusters, with AWS handling scaling, patching, and upgrading.

Combined pattern: ArgoCD deploys ACK resources + KRO compositions via GitOps, providing a single workflow for both infrastructure and applications.

For detailed ArgoCD patterns, see: ArgoCD Patterns Reference

Sources:

• EKS Capabilities Documentation
• AWS Blog — Deep dive: Simplifying resource orchestration with Amazon EKS Capabilities

Detailed References

This skill uses progressive disclosure — essential guidance is in this main file, detailed reference material is loaded on demand:

• Security — IAM, Cluster Access Manager, Pod Identity, IRSA, pod security standards, multi-tenancy, secrets management, data encryption
• Security — Runtime & Network — Runtime threat detection (GuardDuty, seccomp, AppArmor, Falco), network policies, SG for pods, encryption in transit, detective controls
• Security — Supply Chain & Compliance — Image security (SBOMs, attestations, ECR hardening), infrastructure hardening (Bottlerocket, CIS benchmarks), regulatory compliance, incident response
• Networking — VPC CNI modes (secondary IP, prefix delegation, custom networking), subnet/CIDR planning, IPv4 vs IPv6, Security Groups for Pods, IP address management
• Networking — Ingress & DNS — Ingress patterns (ALB, NLB, Gateway API), AWS Load Balancer Controller, service mesh, DNS/CoreDNS tuning, private cluster connectivity
• Reliability & Resiliency — Core — HA patterns, PDBs, health probes, load balancer health checks, lifecycle hooks, topology spread, resource management
• Reliability & Resiliency — Advanced — disaster recovery, zonal shift, deployment strategies, large cluster guidance, chaos engineering, admission-controller topology enforcement
• Surge Readiness — preparing for a known traffic peak (flash sale, marketing push, product launch, seasonal event): spike shape vs magnitude, scheduled pre-scaling, capacity assurance (ODCR, quotas, LB warm-up), load-test realism, graceful degradation, and a descriptive pre-event readiness checklist
• Autoscaling — Autoscaler selection, Cluster Autoscaler (IAM, Spot, overprovisioning, parameter tuning), HPA, VPA, KEDA, CoreDNS autoscaling
• Karpenter — Operational best practices, NodePools, EC2NodeClass, Spot/interruption handling, consolidation, multiple NodePool strategy, cost controls, resource management, private clusters, CoreDNS with Karpenter
• Cluster Upgrades — In-place and blue-green upgrades, pre-upgrade validation, add-on management, API deprecation detection, version skew policy, Bottlerocket updates, rollback procedures
• Cost Optimization — CFM framework, compute/networking/storage cost strategies, observability cost management, Spot, Graviton, tagging, Kubecost
• Scalability — Scaling theory (churn rate, QPS), control plane (APF, config parameters, monitoring), data plane (node sizing, diversity), cluster services (CoreDNS, Metrics Server), workload patterns, IPVS, large-cluster guidance
• Observability — Observability strategy, CloudWatch Container Insights & Application Signals, Prometheus/Grafana, control plane monitoring, network performance monitoring, logging architecture, distributed tracing, GPU/AI-ML observability, detective controls, alerting patterns
• Terraform Examples — terraform-aws-modules/terraform-aws-eks examples, submodules, add-on management, Provisioned Control Plane, EFA, VPC patterns, deployment topologies
• ArgoCD Patterns — ArgoCD architecture, App of Apps, ApplicationSets, GitOps Bridge, multi-cluster patterns (hub-and-spoke, decentralized, hybrid), EKS ArgoCD Capability (managed vs self-managed, migration), ACK/KRO integration, multi-tenant RBAC
• Container Registry — ECR architecture, operating models, image promotion, vulnerability scanning, base image curation, lifecycle policies, pull-through cache, repository creation templates, managed signing (AWS Signer), archival storage class, registry configuration
• EKS Auto Mode — Auto Mode architecture, managed NodePools/NodeClasses, migration from standard EKS, comparison with self-managed Karpenter, limitations and FAQ
• Hybrid & On-Premises Deployments — EKS Hybrid Nodes, EKS Anywhere, and EKS on Outposts: model-selection decision boundary, per-model networking/CIDR/CNI, disconnection behavior, compute & autoscaling support, identity, lifecycle, and a cross-model support matrix

How to use: When you need detailed information on a topic, reference the appropriate guide. Claude will load it on demand.

Sources

• AWS EKS Best Practices Guide
• terraform-aws-modules/terraform-aws-eks