DevOps Kubernetes
Agent Workflow (MANDATORY)
Before ANY manifest or cluster change, spawn in parallel, one spawn_agent call each:
- explore-codebase - find existing manifests, overlays, charts, the target namespace layout
- research-expert - verify the API fields for the cluster's minor at kubernetes.io / gateway-api.sigs.k8s.io
- mcp__context7__query-docs -
/websites/kubernetes_io, /websites/gateway-api_sigs_k8s_io
Then read the cluster read-only (kubectl version, kubectl api-resources, kubectl get crd | grep gateway).
After changes, run sniper.
Versions (checked 2026-11-10)
Kubernetes v1.37.2 / v1.36.6 / v1.35.10 are the latest patches (GitHub release tags, 2026-10-28; https://kubernetes.io/releases/ still listed 1.37.1); 1.34 reached end of life 2026-10-27. Gateway API v1.7.0 (2026-10-29). Each minor deprecates or graduates fields: read the release blog of the target minor (https://kubernetes.io/blog/) before writing manifests for it.
Pinned versions:
kubernetes: "1.37.2 (supported minors 1.37, 1.36, 1.35)"
gateway-api: "1.7.0"
Overview
| Area |
Decision in one line |
| Workloads |
Stateless = Deployment; stable identity/storage = StatefulSet; one per node = DaemonSet; run-to-completion = Job/CronJob |
| Traffic entry |
Gateway API (gateway.networking.k8s.io/v1); Ingress API is frozen; Ingress NGINX is retired |
| Storage |
PVC + StorageClass (WaitForFirstConsumer, allowVolumeExpansion), Retain for data that must survive |
| Config |
ConfigMap for config, Secret for credentials; Secrets are unencrypted in etcd by default |
| Access |
One ServiceAccount per workload, namespaced Role, automountServiceAccountToken: false unless needed |
| Capacity |
Requests on every container, memory limit always, HPA autoscaling/v2 with behavior |
| Isolation |
Default-deny NetworkPolicy per namespace, then explicit allows (needs a CNI that enforces it) |
Critical Rules
- Version first - read the server minor (
kubectl version) and the CRDs present before writing a field; a field that is stable in 1.37 may be beta or absent in 1.35
- Explicit target on every command -
--context <ctx> -n <ns>; never rely on the current context
- Diff and server dry-run before apply -
kubectl diff then kubectl apply --dry-run=server; never kubectl apply --prune (alpha)
- Destructive verbs are proposed, not run -
delete, drain, scale --replicas=0, rollout undo, --force: show the exact effect and blast radius, wait for the user's go (kubectl-safe-ops.md)
- Troubleshoot in order - describe → events → logs → exec/debug → node; no fix before the evidence (troubleshooting-playbook.md)
- Never print a Secret value - names and keys only; values come from the secret store (devops-secrets)
- Never invent an apiVersion or field - cite the kubernetes.io or gateway-api page; unknown = "not verified"
Architecture
k8s/
├── namespace.yaml # PSA labels: pod-security.kubernetes.io/enforce: restricted
├── rbac.yaml # ServiceAccount + Role + RoleBinding (namespaced)
├── networkpolicy.yaml # default-deny + explicit allows
├── config.yaml # ConfigMap (Secrets come from devops-secrets tooling)
├── deployment.yaml # Deployment + Service + PodDisruptionBudget
├── hpa.yaml # autoscaling/v2
└── route.yaml # HTTPRoute attached to the shared Gateway
Reference Guide
Concepts
| Topic |
Reference |
When to Consult |
| Workloads |
workloads-scheduling.md |
Choosing a controller, rollout strategy, PDB, spreading |
| Probes and lifecycle |
probes-lifecycle.md |
startup/readiness/liveness, sidecars, graceful shutdown |
| Services and Gateway API |
services-gateway-api.md |
Exposing an app, leaving Ingress NGINX |
| NetworkPolicy |
network-policy.md |
Namespace isolation, egress control |
| Storage |
storage-csi.md |
PVC sizing, reclaim policy, expansion, access modes |
| Config and Secrets |
config-secrets.md |
Mounting config, rollout on change, Secret handling |
| RBAC |
rbac-serviceaccounts.md |
Least privilege, auth can-i, Pod Security Admission |
| Resources and HPA |
resources-qos-hpa.md |
Requests/limits, QoS, in-place resize, autoscaling |
| Safe kubectl |
kubectl-safe-ops.md |
Any mutating command |
| Troubleshooting |
troubleshooting-playbook.md |
Pending, CrashLoopBackOff, DNS, no endpoints |
Templates
| Template |
When to Use |
| deployment-production.yaml |
Stateless app: probes, limits, restricted securityContext, Service, PDB |
| statefulset-pvc.yaml |
Stateful app with headless Service and per-replica PVC |
| cronjob.yaml |
Scheduled job with time zone, concurrency and history limits |
| gateway-httproute.yaml |
Shared Gateway, HTTPS redirect, app HTTPRoute, ReferenceGrant |
| networkpolicy-default-deny.yaml |
Default deny + DNS + gateway ingress allows |
| rbac-least-privilege.yaml |
ServiceAccount + Role + RoleBinding |
| hpa.yaml |
CPU-based HPA with scale-down stabilization |
Quick Reference
CTX=prod-eu; NS=shop # always explicit
kubectl --context "$CTX" version # server minor
kubectl --context "$CTX" -n "$NS" diff -f k8s/ # exit 0 none, 1 diff, >1 error
kubectl --context "$CTX" -n "$NS" apply --dry-run=server -f k8s/
kubectl --context "$CTX" -n "$NS" apply -f k8s/ # after the user's go
kubectl --context "$CTX" -n "$NS" rollout status deploy/web --timeout=5m
kubectl --context "$CTX" -n "$NS" auth can-i list secrets --as=system:serviceaccount:"$NS":web
Best Practices
DO
- Pin the Gateway API CRD release you install (
standard-install.yaml of a named tag) and match the controller's conformance
- Give every Deployment with more than one replica a PodDisruptionBudget
- Use a startupProbe for slow starters instead of a long liveness
initialDelaySeconds
- Keep manifests in Git, pin images by tag and digest (devops-docker), apply through the pipeline (devops-gitops-packaging)
DON'T
- Start new work on Ingress NGINX (retired) or new Ingress features (API frozen)
- Use
kubectl edit or kubectl patch on objects owned by GitOps - the controller reverts it
- Put CPU limits by reflex - they throttle; decide per workload (resources-qos-hpa.md)
Related skills
- devops-k8s-cluster
- devops-gitops-packaging
- devops-secrets
- devops-observability
- devops-storage-backup
- devops-networking
- devops-docker
- devops-sre-ops
1---2name: devops-kubernetes-23description: Use when writing, reviewing, applying or debugging workloads on an existing Kubernetes cluster - Deployment, StatefulSet, DaemonSet, Job/CronJob, ConfigMap/Secret usage, Services, Gateway API (HTTPRoute) or a migration away from Ingress/Ingress NGINX, PV/PVC/StorageClass, RBAC and ServiceAccounts, requests/limits, probes, HPA, NetworkPolicy, safe kubectl (diff, server dry-run, delete/drain/scale/rollback) and pod troubleshooting (Pending, CrashLoopBackOff, DNS, no endpoints). Not for installing or upgrading clusters, etcd or CNI choice (devops-k8s-cluster), Helm/Kustomize/Argo CD/Flux (devops-gitops-packaging), secret stores (devops-secrets) or a cluster security audit (security-expert).4metadata:5 source: https://skillmd.com/skills/fusengine/devops-kubernetes-26---78<objective>9Covers using a running Kubernetes cluster: workload controllers, configuration,10Services and traffic entry through Gateway API, storage claims, RBAC,11resources/QoS/autoscaling, probes, NetworkPolicy, safe kubectl operation and a12fixed troubleshooting order. Does not cover building or upgrading the cluster,13etcd, CNI selection (devops-k8s-cluster), packaging and GitOps controllers14(devops-gitops-packaging), external secret stores (devops-secrets) or a15security audit (security-expert).16</objective>1718# DevOps Kubernetes1920## Agent Workflow (MANDATORY)2122Before ANY manifest or cluster change, spawn in parallel, one `spawn_agent` call each:23241. **explore-codebase** - find existing manifests, overlays, charts, the target namespace layout252. **research-expert** - verify the API fields for the cluster's minor at kubernetes.io / gateway-api.sigs.k8s.io263. **mcp__context7__query-docs** - `/websites/kubernetes_io`, `/websites/gateway-api_sigs_k8s_io`2728Then read the cluster read-only (`kubectl version`, `kubectl api-resources`, `kubectl get crd | grep gateway`).29After changes, run **sniper**.3031## Versions (checked 2026-11-10)3233Kubernetes v1.37.2 / v1.36.6 / v1.35.10 are the latest patches (GitHub release tags, 2026-10-28; https://kubernetes.io/releases/ still listed 1.37.1); 1.34 reached end of life 2026-10-27. Gateway API v1.7.0 (2026-10-29). Each minor deprecates or graduates fields: read the release blog of the target minor (https://kubernetes.io/blog/) before writing manifests for it.3435Pinned versions:3637```yaml38kubernetes: "1.37.2 (supported minors 1.37, 1.36, 1.35)"39gateway-api: "1.7.0"40```4142## Overview4344| Area | Decision in one line |45|------|----------------------|46| **Workloads** | Stateless = Deployment; stable identity/storage = StatefulSet; one per node = DaemonSet; run-to-completion = Job/CronJob |47| **Traffic entry** | Gateway API (`gateway.networking.k8s.io/v1`); Ingress API is frozen; Ingress NGINX is retired |48| **Storage** | PVC + StorageClass (`WaitForFirstConsumer`, `allowVolumeExpansion`), `Retain` for data that must survive |49| **Config** | ConfigMap for config, Secret for credentials; Secrets are unencrypted in etcd by default |50| **Access** | One ServiceAccount per workload, namespaced Role, `automountServiceAccountToken: false` unless needed |51| **Capacity** | Requests on every container, memory limit always, HPA `autoscaling/v2` with `behavior` |52| **Isolation** | Default-deny NetworkPolicy per namespace, then explicit allows (needs a CNI that enforces it) |5354## Critical Rules55561. **Version first** - read the server minor (`kubectl version`) and the CRDs present before writing a field; a field that is stable in 1.37 may be beta or absent in 1.35572. **Explicit target on every command** - `--context <ctx> -n <ns>`; never rely on the current context583. **Diff and server dry-run before apply** - `kubectl diff` then `kubectl apply --dry-run=server`; never `kubectl apply --prune` (alpha)594. **Destructive verbs are proposed, not run** - `delete`, `drain`, `scale --replicas=0`, `rollout undo`, `--force`: show the exact effect and blast radius, wait for the user's go ([kubectl-safe-ops.md](references/kubectl-safe-ops.md))605. **Troubleshoot in order** - describe → events → logs → exec/debug → node; no fix before the evidence ([troubleshooting-playbook.md](references/troubleshooting-playbook.md))616. **Never print a Secret value** - names and keys only; values come from the secret store (devops-secrets)627. **Never invent an apiVersion or field** - cite the kubernetes.io or gateway-api page; unknown = "not verified"6364## Architecture6566```67k8s/68├── namespace.yaml # PSA labels: pod-security.kubernetes.io/enforce: restricted69├── rbac.yaml # ServiceAccount + Role + RoleBinding (namespaced)70├── networkpolicy.yaml # default-deny + explicit allows71├── config.yaml # ConfigMap (Secrets come from devops-secrets tooling)72├── deployment.yaml # Deployment + Service + PodDisruptionBudget73├── hpa.yaml # autoscaling/v274└── route.yaml # HTTPRoute attached to the shared Gateway75```7677## Reference Guide7879### Concepts8081| Topic | Reference | When to Consult |82|-------|-----------|-----------------|83| **Workloads** | [workloads-scheduling.md](references/workloads-scheduling.md) | Choosing a controller, rollout strategy, PDB, spreading |84| **Probes and lifecycle** | [probes-lifecycle.md](references/probes-lifecycle.md) | startup/readiness/liveness, sidecars, graceful shutdown |85| **Services and Gateway API** | [services-gateway-api.md](references/services-gateway-api.md) | Exposing an app, leaving Ingress NGINX |86| **NetworkPolicy** | [network-policy.md](references/network-policy.md) | Namespace isolation, egress control |87| **Storage** | [storage-csi.md](references/storage-csi.md) | PVC sizing, reclaim policy, expansion, access modes |88| **Config and Secrets** | [config-secrets.md](references/config-secrets.md) | Mounting config, rollout on change, Secret handling |89| **RBAC** | [rbac-serviceaccounts.md](references/rbac-serviceaccounts.md) | Least privilege, `auth can-i`, Pod Security Admission |90| **Resources and HPA** | [resources-qos-hpa.md](references/resources-qos-hpa.md) | Requests/limits, QoS, in-place resize, autoscaling |91| **Safe kubectl** | [kubectl-safe-ops.md](references/kubectl-safe-ops.md) | Any mutating command |92| **Troubleshooting** | [troubleshooting-playbook.md](references/troubleshooting-playbook.md) | Pending, CrashLoopBackOff, DNS, no endpoints |9394### Templates9596| Template | When to Use |97|----------|-------------|98| [deployment-production.yaml](references/templates/deployment-production.yaml) | Stateless app: probes, limits, restricted securityContext, Service, PDB |99| [statefulset-pvc.yaml](references/templates/statefulset-pvc.yaml) | Stateful app with headless Service and per-replica PVC |100| [cronjob.yaml](references/templates/cronjob.yaml) | Scheduled job with time zone, concurrency and history limits |101| [gateway-httproute.yaml](references/templates/gateway-httproute.yaml) | Shared Gateway, HTTPS redirect, app HTTPRoute, ReferenceGrant |102| [networkpolicy-default-deny.yaml](references/templates/networkpolicy-default-deny.yaml) | Default deny + DNS + gateway ingress allows |103| [rbac-least-privilege.yaml](references/templates/rbac-least-privilege.yaml) | ServiceAccount + Role + RoleBinding |104| [hpa.yaml](references/templates/hpa.yaml) | CPU-based HPA with scale-down stabilization |105106## Quick Reference107108```bash109CTX=prod-eu; NS=shop # always explicit110kubectl --context "$CTX" version # server minor111kubectl --context "$CTX" -n "$NS" diff -f k8s/ # exit 0 none, 1 diff, >1 error112kubectl --context "$CTX" -n "$NS" apply --dry-run=server -f k8s/113kubectl --context "$CTX" -n "$NS" apply -f k8s/ # after the user's go114kubectl --context "$CTX" -n "$NS" rollout status deploy/web --timeout=5m115kubectl --context "$CTX" -n "$NS" auth can-i list secrets --as=system:serviceaccount:"$NS":web116```117118## Best Practices119120### DO121- Pin the Gateway API CRD release you install (`standard-install.yaml` of a named tag) and match the controller's conformance122- Give every Deployment with more than one replica a PodDisruptionBudget123- Use a startupProbe for slow starters instead of a long liveness `initialDelaySeconds`124- Keep manifests in Git, pin images by tag and digest (devops-docker), apply through the pipeline (devops-gitops-packaging)125126### DON'T127- Start new work on Ingress NGINX (retired) or new Ingress features (API frozen)128- Use `kubectl edit` or `kubectl patch` on objects owned by GitOps - the controller reverts it129- Put CPU limits by reflex - they throttle; decide per workload ([resources-qos-hpa.md](references/resources-qos-hpa.md))130131---132133## Related skills134135- devops-k8s-cluster136- devops-gitops-packaging137- devops-secrets138- devops-observability139- devops-storage-backup140- devops-networking141- devops-docker142- devops-sre-ops