Kubernetes

Kubernetes Manifest Guide

Write production-grade Kubernetes manifests that are secure, observable, and resilient by default. Every manifest should be deployable with a single kubectl apply -f k8s/ and produce a working system.

Core Principles

1. Declarative over imperative — define desired state in YAML; let the controller loop converge. Never rely on manual kubectl sequences for production state.
2. Namespace isolation — every project gets its own namespace. All resources reference it explicitly so kubectl apply is idempotent and scoped.
3. Least privilege — containers run as non-root, Secrets hold credentials, ConfigMaps hold non-sensitive config. Never embed secrets in manifests committed to source control (use placeholders + kubectl create secret).
4. Resource budgeting — every container declares requests (scheduling guarantee) and limits (hard ceiling). This prevents noisy-neighbor starvation and enables HPA.
5. Health-first deployments — every long-running container has a liveness probe (restart on hang) and a readiness probe (stop routing on dependency failure). Probes must be lightweight and side-effect-free.
6. Persistent state is explicit — databases and caches use PVCs with named claims. Stateless services never mount persistent volumes.

Manifest Fundamentals

API Version & Kind

Always use the stable API version for each resource:

# Workloads
apiVersion: apps/v1          # Deployment, StatefulSet, DaemonSet
# Core resources
apiVersion: v1               # Service, ConfigMap, Secret, Namespace, PVC
# Scaling
apiVersion: autoscaling/v2   # HorizontalPodAutoscaler
# Ingress
apiVersion: networking.k8s.io/v1  # Ingress

Metadata Convention

Every resource needs consistent metadata for filtering and management:

metadata:
  name: <resource-name>
  namespace: <project-namespace>    # Always explicit — never rely on context
  labels:
    app: <service-name>             # Selector target
    component: <role>               # backend, frontend, database, cache
    part-of: <project-name>         # Groups all project resources

Label Selectors

Deployments and Services find pods via label selectors. The golden rule: spec.selector.matchLabels must be a subset of spec.template.metadata.labels.

# Deployment
spec:
  selector:
    matchLabels:
      app: my-api                # ← Must match pod template labels
  template:
    metadata:
      labels:
        app: my-api              # ← Matched by selector
        component: backend       # ← Extra labels are fine

# Service targeting the same pods
spec:
  selector:
    app: my-api                  # ← Same label selects the same pods

Namespace

Isolate project resources. Create the namespace first — everything else references it.

apiVersion: v1
kind: Namespace
metadata:
  name: my-project
  labels:
    app: my-project

Apply order: namespace → config → storage → workloads → scaling.

ConfigMap & Secret

ConfigMap — Non-sensitive configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: my-project
data:
  LOG_LEVEL: "info"
  DATABASE_HOST: "postgres"
  CACHE_URL: "redis://redis:6379"

Secret — Sensitive credentials

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: my-project
type: Opaque
data:
  DATABASE_PASSWORD: ""       # Base64: echo -n 'value' | base64
  API_KEY: ""                 # Never commit real values

Mounting in pods — use envFrom to inject all keys as environment variables:

spec:
  containers:
    - name: app
      envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets

For composite values (like a DATABASE_URL that embeds a password), use env with variable substitution:

env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: DATABASE_PASSWORD
  - name: DATABASE_URL
    value: "postgresql://user:$(DATABASE_PASSWORD)@postgres:5432/mydb"

K8s resolves $(VAR_NAME) from previously defined env vars in the same container spec. Order matters — the referenced variable must appear first.

PersistentVolumeClaim

PVCs decouple storage from pod lifecycle. Data survives pod restarts, image pulls, and rescheduling.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-pvc
  namespace: my-project
spec:
  accessModes:
    - ReadWriteOnce           # Single-node write (standard for databases)
  resources:
    requests:
      storage: 1Gi

Access modes:

Mount in a deployment:

spec:
  containers:
    - volumeMounts:
        - name: db-storage
          mountPath: /var/lib/postgresql/data
  volumes:
    - name: db-storage
      persistentVolumeClaim:
        claimName: db-pvc

Deployments

Resource Requests & Limits

Every container must declare resources. Without them, HPA cannot calculate utilization and the scheduler cannot make informed placement decisions.

resources:
  requests:              # Guaranteed minimum — scheduler reserves this
    cpu: 100m            # 100 millicores = 0.1 CPU core
    memory: 256Mi        # 256 MiB RAM
  limits:                # Hard ceiling — OOMKill if memory exceeded
    cpu: 500m
    memory: 512Mi

Sizing guidelines:

These are starting points. For production, profile actual usage and set requests to P50, limits to P99.

Health Probes

Probes serve fundamentally different purposes — choosing the wrong one causes cascading failures.

Liveness probe — "Is this process alive?" Restart the pod if it fails. Use for detecting deadlocks, infinite loops, or zombie processes. Keep it cheap — no dependency checks.

livenessProbe:
  httpGet:
    path: /healthz             # Or /health/live — lightweight, no deps
    port: 8000
  initialDelaySeconds: 10      # Wait for app startup
  periodSeconds: 10            # Check every 10s
  timeoutSeconds: 5            # Fail if no response in 5s
  failureThreshold: 3          # Restart after 3 consecutive failures

Readiness probe — "Can this instance serve traffic?" Remove from Service endpoints if it fails. Use for dependency checks (database, cache). The pod stays running — it just stops receiving traffic until recovery.

readinessProbe:
  httpGet:
    path: /ready               # Or /health/ready — checks dependencies
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 5             # Check frequently — fast recovery
  timeoutSeconds: 5
  failureThreshold: 2          # Remove quickly on failure

Startup probe — "Has the app finished starting?" Disables liveness/readiness until the app is ready. Use for slow-starting applications (JVM warmup, large model loading).

startupProbe:
  httpGet:
    path: /healthz
    port: 8000
  periodSeconds: 5
  failureThreshold: 30         # 30 × 5s = 150s max startup time

Critical mistakes to avoid:

• Never put dependency checks in the liveness probe — if the database goes down, you don't want every API pod restarting (cascade failure)
• Always set initialDelaySeconds — probing before the app starts causes unnecessary restarts
• Keep probe endpoints fast (< 200ms) — slow probes cause false positives

Init Containers

Run setup tasks before the main container starts. Common uses: database migrations, config generation, waiting for dependencies.

spec:
  initContainers:
    - name: run-migrations
      image: my-app:latest           # Reuse the app image
      command: ["bash", "scripts/migrate.sh"]
      envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets
  containers:
    - name: app
      # ... main container starts after init completes

Init container rules:

• Runs to completion before main container starts
• If it fails, the pod restarts (respects restartPolicy)
• Has access to the same volumes, ConfigMaps, and Secrets as main container
• Scripts must be idempotent — they run on every pod start

Rolling Update Strategy

Control how deployments roll out new versions:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1              # Create 1 extra pod during update
      maxUnavailable: 0        # Never reduce below desired replicas

maxUnavailable: 0 ensures zero-downtime deploys — new pods must pass readiness before old pods are terminated.

For single-instance databases, use strategy: Recreate to prevent dual-write.

Services

Services provide stable DNS names and load balancing for pods.

ClusterIP (Internal)

Default type. Accessible within the cluster via DNS: <service-name>.<namespace>.svc.cluster.local (or just <service-name> within the same namespace).

apiVersion: v1
kind: Service
metadata:
  name: my-api
  namespace: my-project
spec:
  type: ClusterIP
  selector:
    app: my-api
  ports:
    - name: http
      port: 8000
      targetPort: 8000
      protocol: TCP

NodePort (External Access for Dev)

Exposes on each node's IP at a static port (30000–32767). Use for local dev when you need external access without an Ingress controller.

apiVersion: v1
kind: Service
metadata:
  name: my-web
  namespace: my-project
spec:
  type: NodePort
  selector:
    app: my-web
  ports:
    - name: http
      port: 3000
      targetPort: 3000
      nodePort: 30000         # Optional: omit for auto-assignment

When to use which

HorizontalPodAutoscaler

Auto-scale workloads based on observed metrics. Requires resource requests on target pods (HPA calculates utilization as current / request).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-api-hpa
  namespace: my-project
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-api
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70    # Scale up when avg CPU > 70%

Behavior tuning (prevent flapping):

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300   # 5 minutes cooldown
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

Scale up aggressively to handle spikes; scale down conservatively to avoid oscillation.

Best Practices

1. Always set a namespace explicitly — never depend on kubectl context. Manifests should be self-contained and apply correctly regardless of the user's current context.

2. Never commit real secrets — use empty placeholders in manifests and create real secrets with kubectl create secret generic. Add secret files to .gitignore if they ever contain real values.

3. Match probe type to failure mode — liveness for process health (no deps), readiness for dependency health (DB, cache). Mixing them causes cascading restarts.

4. Set resource requests AND limits on every container — requests enable scheduling; limits prevent runaway consumption. Without requests, HPA cannot function.

5. Use envFrom over individual env entries — cleaner manifests, easier to add new config without touching deployments.

6. Make init containers idempotent — they run on every pod start, not just the first time. Use IF NOT EXISTS in SQL, check-then-create patterns.

7. PVCs for stateful services only — databases and caches get PVCs. API servers and web frontends should be stateless and disposable.

8. Label everything consistently — app, component, and part-of labels enable kubectl get pods -l app=my-api and service discovery.

9. Order manifests for dependency — namespace first, then config/secrets, then storage, then workloads, then scaling. Use file naming (00-namespace.yml, 10-config.yml) or kustomize for ordering.

10. Test with kubectl apply --dry-run=client — catches syntax errors and missing fields before hitting the cluster. Follow with kubectl diff to preview changes against the live state.

When to Read Reference Files

Example Templates