DevOps Kubernetes
Agent Workflow (MANDATORY)
Before ANY manifest or cluster change, spawn in parallel, one Agent call each with a name:
- fuse-ai-pilot:explore-codebase - find existing manifests, overlays, charts, the target namespace layout
- fuse-ai-pilot:research-expert - verify the API fields for the cluster's minor at kubernetes.io / gateway-api.sigs.k8s.io
- mcp__context7__query-docs -
/websites/kubernetes_io,/websites/gateway-api_sigs_k8s_io
Then read the cluster read-only (kubectl version, kubectl api-resources, kubectl get crd | grep gateway).
After changes, run fuse-ai-pilot:sniper.
Versions (checked 2026-11-10)
Kubernetes v1.37.2 / v1.36.6 / v1.35.10 are the latest patches (GitHub release tags, 2026-10-28; https://kubernetes.io/releases/ still listed 1.37.1); 1.34 reached end of life 2026-10-27. Gateway API v1.7.0 (2026-10-29). Each minor deprecates or graduates fields: read the release blog of the target minor (https://kubernetes.io/blog/) before writing manifests for it.
Overview
| Area | Decision in one line |
|---|---|
| Workloads | Stateless = Deployment; stable identity/storage = StatefulSet; one per node = DaemonSet; run-to-completion = Job/CronJob |
| Traffic entry | Gateway API (gateway.networking.k8s.io/v1); Ingress API is frozen; Ingress NGINX is retired |
| Storage | PVC + StorageClass (WaitForFirstConsumer, allowVolumeExpansion), Retain for data that must survive |
| Config | ConfigMap for config, Secret for credentials; Secrets are unencrypted in etcd by default |
| Access | One ServiceAccount per workload, namespaced Role, automountServiceAccountToken: false unless needed |
| Capacity | Requests on every container, memory limit always, HPA autoscaling/v2 with behavior |
| Isolation | Default-deny NetworkPolicy per namespace, then explicit allows (needs a CNI that enforces it) |
Critical Rules
- Version first - read the server minor (
kubectl version) and the CRDs present before writing a field; a field that is stable in 1.37 may be beta or absent in 1.35 - Explicit target on every command -
--context <ctx> -n <ns>; never rely on the current context - Diff and server dry-run before apply -
kubectl diffthenkubectl apply --dry-run=server; neverkubectl apply --prune(alpha) - Destructive verbs are proposed, not run -
delete,drain,scale --replicas=0,rollout undo,--force: show the exact effect and blast radius, wait for the user's go (kubectl-safe-ops.md) - Troubleshoot in order - describe → events → logs → exec/debug → node; no fix before the evidence (troubleshooting-playbook.md)
- Never print a Secret value - names and keys only; values come from the secret store (devops-secrets)
- Never invent an apiVersion or field - cite the kubernetes.io or gateway-api page; unknown = "not verified"
Architecture
k8s/
├── namespace.yaml # PSA labels: pod-security.kubernetes.io/enforce: restricted
├── rbac.yaml # ServiceAccount + Role + RoleBinding (namespaced)
├── networkpolicy.yaml # default-deny + explicit allows
├── config.yaml # ConfigMap (Secrets come from devops-secrets tooling)
├── deployment.yaml # Deployment + Service + PodDisruptionBudget
├── hpa.yaml # autoscaling/v2
└── route.yaml # HTTPRoute attached to the shared Gateway
Reference Guide
Concepts
| Topic | Reference | When to Consult |
|---|---|---|
| Workloads | workloads-scheduling.md | Choosing a controller, rollout strategy, PDB, spreading |
| Probes and lifecycle | probes-lifecycle.md | startup/readiness/liveness, sidecars, graceful shutdown |
| Services and Gateway API | services-gateway-api.md | Exposing an app, leaving Ingress NGINX |
| NetworkPolicy | network-policy.md | Namespace isolation, egress control |
| Storage | storage-csi.md | PVC sizing, reclaim policy, expansion, access modes |
| Config and Secrets | config-secrets.md | Mounting config, rollout on change, Secret handling |
| RBAC | rbac-serviceaccounts.md | Least privilege, auth can-i, Pod Security Admission |
| Resources and HPA | resources-qos-hpa.md | Requests/limits, QoS, in-place resize, autoscaling |
| Safe kubectl | kubectl-safe-ops.md | Any mutating command |
| Troubleshooting | troubleshooting-playbook.md | Pending, CrashLoopBackOff, DNS, no endpoints |
Templates
| Template | When to Use |
|---|---|
| deployment-production.yaml | Stateless app: probes, limits, restricted securityContext, Service, PDB |
| statefulset-pvc.yaml | Stateful app with headless Service and per-replica PVC |
| cronjob.yaml | Scheduled job with time zone, concurrency and history limits |
| gateway-httproute.yaml | Shared Gateway, HTTPS redirect, app HTTPRoute, ReferenceGrant |
| networkpolicy-default-deny.yaml | Default deny + DNS + gateway ingress allows |
| rbac-least-privilege.yaml | ServiceAccount + Role + RoleBinding |
| hpa.yaml | CPU-based HPA with scale-down stabilization |
Quick Reference
CTX=prod-eu; NS=shop # always explicit
kubectl --context "$CTX" version # server minor
kubectl --context "$CTX" -n "$NS" diff -f k8s/ # exit 0 none, 1 diff, >1 error
kubectl --context "$CTX" -n "$NS" apply --dry-run=server -f k8s/
kubectl --context "$CTX" -n "$NS" apply -f k8s/ # after the user's go
kubectl --context "$CTX" -n "$NS" rollout status deploy/web --timeout=5m
kubectl --context "$CTX" -n "$NS" auth can-i list secrets --as=system:serviceaccount:"$NS":web
Best Practices
DO
- Pin the Gateway API CRD release you install (
standard-install.yamlof a named tag) and match the controller's conformance - Give every Deployment with more than one replica a PodDisruptionBudget
- Use a startupProbe for slow starters instead of a long liveness
initialDelaySeconds - Keep manifests in Git, pin images by tag and digest (devops-docker), apply through the pipeline (devops-gitops-packaging)
DON'T
- Start new work on Ingress NGINX (retired) or new Ingress features (API frozen)
- Use
kubectl editorkubectl patchon objects owned by GitOps - the controller reverts it - Put CPU limits by reflex - they throttle; decide per workload (resources-qos-hpa.md)
