# Kubernetes Basics > Kubernetes cluster management — resources, networking, storage, Helm, troubleshooting. Use when writing Kubernetes manifests, running kubectl or helm, debugging pods and services, or configuring ingress, DNS, storage, or network policies. - Skill: `projectious-work/kubernetes-basics` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds add projectious-work/kubernetes-basics` - Raw SKILL.md: https://api.skillmd.com/api/skills/projectious-work/kubernetes-basics/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: projectious-work (https://skillmd.com/u/projectious-work) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/projectious-work/kubernetes-basics --- # Kubernetes Basics ## Intro Kubernetes is a declarative system: you write manifests, the API server stores them in etcd, and controllers reconcile reality toward that desired state. Confirm the active context, prefer declarative `apply` over imperative commands, and always check events and logs before poking at YAML. ## Overview ### Cluster context Always confirm the active cluster and namespace before making changes: ```bash kubectl config current-context kubectl config get-contexts kubectl get ns ``` Switch context/namespace explicitly rather than relying on defaults: ```bash kubectl config use-context kubectl config set-context --current --namespace= ``` ### Core resources Use Deployments for stateless workloads and StatefulSets for stateful ones. Never create bare Pods in production — they are not rescheduled on failure. - **Pod** — smallest deployable unit, one or more containers - **Deployment** — manages ReplicaSets, handles rolling updates - **StatefulSet** — stable network identity and persistent storage per replica - **DaemonSet** — one pod per node (logging, monitoring agents) - **Job / CronJob** — run-to-completion and scheduled tasks ### Networking Services expose pods internally or externally: - **ClusterIP** (default) — internal only - **NodePort** — exposes on each node's IP at a static port - **LoadBalancer** — provisions an external LB (cloud providers) Ingress routes external HTTP/HTTPS to services by host/path. Always set `ingressClassName`. Pods resolve services via DNS at `..svc.cluster.local`. Network policies default to allow-all. Define explicit ingress/egress rules to restrict traffic between namespaces or pods. ### Storage - **PersistentVolume (PV)** — cluster-level storage resource - **PersistentVolumeClaim (PVC)** — namespace-scoped request for storage - **StorageClass** — dynamic provisioning template Use `storageClassName` in PVCs. For StatefulSets, use `volumeClaimTemplates` to create per-replica PVCs automatically. Always set `accessModes` and `resources.requests.storage`. ### Configuration - **ConfigMap** — non-sensitive key-value config; mount as files or env vars - **Secret** — base64-encoded sensitive data; same mount options Prefer mounting as volumes over env vars for config files. Use `kubectl create secret generic` for quick creation. In production, use an external secrets operator or sealed-secrets. ### Helm Helm manages templated releases: ```bash helm repo add helm repo update helm search repo helm install -n --create-namespace -f values.yaml helm upgrade -n -f values.yaml helm rollback -n helm list -n helm uninstall -n ``` Inspect before installing: `helm template` to render, `helm show values` for defaults. Pin chart versions in CI with `--version`. ### Troubleshooting order When debugging, start broad and narrow down: 1. `kubectl get events --sort-by=.lastTimestamp` — cluster events 2. `kubectl describe pod ` — scheduling, image pull, probes 3. `kubectl logs [-c container] [--previous]` — app logs 4. `kubectl exec -it -- sh` — interactive shell 5. `kubectl top pod` / `kubectl top node` — resource usage ### Applying changes safely Always `kubectl diff` before `kubectl apply` to preview changes. Prefer declarative `apply` over imperative `create`/`edit`. Use `--dry-run=client -o yaml` to validate without submitting. For production rollouts, watch with `kubectl rollout status` and reach for `kubectl rollout undo` if something goes wrong. ## Gotchas Agent-specific failure modes — provider-neutral pause-and-self-check items: - **Not confirming the active context before making changes.** `kubectl apply -f` operates against whichever cluster context is currently active. Running a destructive operation in production because the context was left pointing at prod from a previous debugging session is among the most common Kubernetes incidents. Always run `kubectl config current-context` before any write operation, and use a tool or alias that shows the context in the prompt. - **Bare Pods in production instead of Deployments.** A bare Pod that crashes is not rescheduled — it stays in `Error` or `CrashLoopBackOff` until manually deleted and recreated. A Deployment's ReplicaSet controller reschedules it automatically and supports rolling updates and rollbacks. Never create bare Pods for any workload that must stay up. - **Missing readiness probes on Services.** A Service will send traffic to all pods that match its selector, including pods that are starting, initializing, or currently unhealthy but have not yet been killed by the liveness probe. Without a readiness probe, traffic hits unready pods and users see errors. The liveness probe kills bad pods; the readiness probe keeps traffic away from them until they are ready. - **No resource requests or limits.** Without `resources.requests`, the scheduler has no signal for node placement and a noisy pod can be scheduled on an already-overloaded node. Without `resources.limits`, a memory-leaking pod can OOMKill neighboring pods on the same node. Set both on every container; start with generous values, observe with `kubectl top`, then tighten. - **`latest` image tags in Deployment manifests.** A Deployment with `image: myapp:latest` will pull whatever `latest` resolves to at each restart — silently deploying a different version than the one you tested. Always pin to a specific immutable tag (typically a Git SHA or build number) so rollouts are deterministic and `kubectl rollout undo` restores the exact previous version. - **Blind `kubectl apply` without reviewing `kubectl diff` first.** `kubectl apply` without a preceding `kubectl diff` is a production change made without preview. Fields set out-of-band (via `kubectl edit`, autoscalers, or admission webhooks) may be silently overwritten. Always `kubectl diff -f manifest.yaml` before `kubectl apply -f manifest.yaml` in production. - **Using `-target` or imperative `kubectl edit` as a normal workflow.** Imperative edits (`kubectl edit`, `kubectl scale`) and `-target` applies produce configuration that diverges from the version-controlled manifests. The next declarative apply will overwrite the change without warning. All production changes must go through version-controlled manifests applied via CI/CD. ## Full reference ### Control plane architecture ``` kubectl apply -f deployment.yaml | v API Server: validates, stores in etcd | v Deployment controller -> ReplicaSet -> Pod objects (Pending) | v Scheduler: assigns Pods to nodes (filtering + scoring) | v kubelet: pulls image, starts containers on the node | v kube-proxy: updates iptables/IPVS rules for Service endpoints ``` Components: - **API server** — front door; validates and persists to etcd - **etcd** — distributed KV store holding all cluster state; HA with 3 or 5 members; back up with `etcdctl snapshot save` - **Scheduler** — filters then scores nodes for unscheduled pods - **Controller manager** — runs reconcile loops (ReplicaSet, Deployment, Node, Job, Endpoint, ServiceAccount controllers) - **kubelet** — node agent; manages Pod lifecycle via the CRI - **kube-proxy** — programs iptables/IPVS/nftables for Services - **Container runtime** — containerd or CRI-O; Docker Engine was removed as a direct runtime in v1.24 ### Resource YAML cheatsheet **Deployment with resources, probes, config, storage:** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 template: metadata: labels: app: myapp spec: containers: - name: app image: myapp:1.0.0 ports: - containerPort: 8080 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 3 periodSeconds: 5 envFrom: - configMapRef: name: myapp-config volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: myapp-data ``` **Service types:** ```yaml apiVersion: v1 kind: Service metadata: name: myapp spec: selector: app: myapp ports: - port: 80 targetPort: 8080 --- apiVersion: v1 kind: Service metadata: name: myapp-lb spec: type: LoadBalancer selector: app: myapp ports: - port: 443 targetPort: 8080 ``` **Ingress with TLS:** ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: myapp spec: ingressClassName: nginx tls: - hosts: [myapp.example.com] secretName: myapp-tls rules: - host: myapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: myapp port: number: 80 ``` **ConfigMap, Secret, PVC:** ```yaml apiVersion: v1 kind: ConfigMap metadata: name: myapp-config data: APP_ENV: production LOG_LEVEL: info --- apiVersion: v1 kind: Secret metadata: name: myapp-secret type: Opaque stringData: DB_PASSWORD: s3cret --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: myapp-data spec: accessModes: [ReadWriteOnce] storageClassName: standard resources: requests: storage: 10Gi ``` **Job and CronJob:** ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: nightly-backup spec: schedule: "0 2 * * *" concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: backup image: myapp:1.0.0 command: ["./backup.sh"] ``` ### kubectl cheatsheet ```bash # Viewing kubectl get pods -n -o wide kubectl get all -n kubectl describe pod -n kubectl get events --sort-by=.lastTimestamp # Applying kubectl apply -f manifest.yaml kubectl apply -f ./manifests/ -R kubectl diff -f manifest.yaml kubectl create secret generic db \ --from-literal=password=s3cret -n # Debugging kubectl logs -n kubectl logs -c --previous kubectl logs -l app=myapp -n --tail=100 kubectl exec -it -n -- sh kubectl port-forward svc/myapp 8080:80 -n kubectl top pod -n kubectl top node # Rollouts kubectl scale deploy myapp --replicas=5 -n kubectl rollout status deploy/myapp -n kubectl rollout history deploy/myapp -n kubectl rollout undo deploy/myapp -n kubectl rollout restart deploy/myapp -n # Deleting kubectl delete -f manifest.yaml kubectl delete pod --grace-period=0 --force ``` ### Troubleshooting patterns **Pod Pending (not scheduled).** Read the events: ```bash kubectl describe pod -n # look at Events section ``` | Event message | Cause | Fix | |---|---|---| | `Insufficient cpu/memory` | Node full | Scale cluster, reduce requests | | `node(s) had taint` | Taint/toleration mismatch | Add toleration or remove taint | | `no persistent volumes available` | PVC cannot bind | Check PV, StorageClass, access modes | | `didn't match Pod's node affinity/selector` | Scheduling constraint | Fix selector or label nodes | **CrashLoopBackOff.** Get logs from the crashed container: ```bash kubectl logs -n --previous kubectl get pod -n \ -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' ``` Exit code 137 is OOMKilled — raise the memory limit. Exit code 1 with no logs usually means a bad `command`/`args`. A failing liveness probe restarts the container on a loop. **ImagePullBackOff.** Almost always a wrong tag, a private registry without `imagePullSecrets`, or Docker Hub rate limiting: ```bash kubectl create secret docker-registry regcred \ --docker-server=ghcr.io \ --docker-username= \ --docker-password= \ -n ``` Reference it in the pod spec via `spec.imagePullSecrets`. **Service not reachable.** Endpoints tell you if the selector matches any pods: ```bash kubectl get svc -n kubectl get endpoints -n # Empty endpoints = selector mismatch OR no pods are Ready kubectl get pods -n --show-labels kubectl run curl --image=curlimages/curl --restart=Never --rm -it -- \ curl -v http://..svc.cluster.local: ``` Check: selector labels match pod labels exactly, `targetPort` matches the container's listening port, readiness probe is passing, NetworkPolicy is not blocking. **DNS issues inside the cluster.** ```bash kubectl run dnstest --image=busybox:1.36 --restart=Never --rm -it -- \ nslookup ..svc.cluster.local kubectl get pods -n kube-system -l k8s-app=kube-dns kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 ``` `ndots:5` can cause slow external lookups — use a fully-qualified name with a trailing dot to force a direct lookup. **Resource limits and OOM.** - `requests` = typical usage, used for scheduling - `limits` = maximum allowed; memory overrun is OOMKill, CPU overrun is throttling - Start with generous limits, monitor with `kubectl top`, then tighten - CPU limits are often best omitted — throttling is less harmful than OOMKill ```bash kubectl top pod -n --containers kubectl get pods -n -o json | \ jq '.items[] | select(.status.containerStatuses[]?.lastState.terminated.reason=="OOMKilled") | .metadata.name' ``` **Node pressure and eviction.** ```bash kubectl describe node | grep -A 5 Conditions # Common: MemoryPressure, DiskPressure, PIDPressure kubectl get pods -n --field-selector=status.phase=Failed | grep Evicted ``` ### Helm upgrade with rollback safety ```bash helm upgrade myrelease mychart/app -n prod -f prod-values.yaml \ --atomic --timeout 5m # --atomic auto-rolls back on failure helm history myrelease -n prod ``` ### Debug containers and RBAC checks ```bash # Ephemeral debug container (K8s 1.23+) kubectl debug -it -n --image=busybox:1.36 --target= # Copy a running pod kubectl debug -n --copy-to=debug-pod --container=app -- sh # RBAC checks kubectl auth can-i create deployments -n kubectl auth can-i '*' '*' --all-namespaces kubectl auth can-i get pods --as=system:serviceaccount:: ``` ### Anti-patterns - **Bare Pods in production** — use Deployments or StatefulSets - **Missing readiness probes** — Services will send traffic to pods that aren't ready - **No resource requests/limits** — scheduler has nothing to go on, noisy neighbors can starve you - **Hardcoded namespace in manifests** — pass via `-n` or kustomize - **`latest` image tags** — rollouts become non-deterministic - **Ignoring `kubectl diff`** — blind `apply` can silently remove fields that were set out-of-band