Kubernetes Statefulset

Kubernetes StatefulSet Manager

Implements apps/v1 StatefulSet manifests for stateful workloads that require stable network identities, ordered deployment and scaling, and persistent storage. When loaded, the model generates production-grade StatefulSet resources with headless service binding, ordered pod management, and volume claim templates.

TL;DR Checklist

• Always pair StatefulSet with a headless service (clusterIP: None)
• Define volumeClaimTemplates in the StatefulSet spec for persistent storage
• Use podManagementPolicy: OrderedReady for ordered scaling (default — verify explicitly)
• Set serviceName to reference the headless service — this enables stable DNS names
• Never use StatefulSet for stateless workloads — use Deployment instead
• Verify PVCs persist after pod deletion — stateful workloads depend on data retention

When to Use

Use this skill when:

• Running a database (MySQL, PostgreSQL, MongoDB, Cassandra) on Kubernetes
• Deploying distributed systems requiring stable hostnames and ordered member addition (etcd, Kafka, ZooKeeper)
• Managing message brokers with partitioned leader election (NATS, RabbitMQ with clustering)
• Implementing application layers that need predictable pod naming and DNS entries
• Orchestrating distributed systems where startup/shutdown order matters

When NOT to Use

Avoid this skill for:

• Stateless web applications or API services — use kubernetes-deployment instead
• Short-lived batch processing jobs — use kubernetes-jobs or kubernetes-cronjob instead
• Applications that do not need stable network identities or persistent storage
• Workloads where scaling up and down must happen in parallel without ordering

Core Workflow

1. Create Headless Service — Define a Service with clusterIP: None that selects pods by the StatefulSet's label selector. Checkpoint: The service spec.selector must match spec.template.metadata.labels of the StatefulSet exactly.

2. Define StatefulSet Spec — Create the StatefulSet with serviceName, replicas, podManagementPolicy, updateStrategy, and volumeClaimTemplates. Checkpoint: serviceName must reference the headless service created in step 1.

3. Configure Volume Claim Templates — Define volumeClaimTemplates with accessModes, storageClassName, and resources.requests.storage. Checkpoint: Each pod in the StatefulSet gets a uniquely numbered PVC (e.g., data-<statefulset>-0, data-<statefulset>-1).

4. Set Update Strategy — Choose RollingUpdate (default) with partition for progressive rollouts, or OnDelete for manual control. Checkpoint: With partition, only pods with ordinal ≥ partition get updated — verify ordinal numbering.

5. Validate StatefulSet Creation — Apply manifests and verify pods create in order (0, 1, 2...) and are deleted in reverse order (N, N-1, ..., 0). Checkpoint: Check DNS names: <pod-name>.<headless-service-name>.<namespace>.svc.cluster.local.

6. Verify Persistent Storage — Confirm PVCs persist after pod deletion and are reattached to the correct pod by ordinal. Checkpoint: Run kubectl get pvc -l app=<statefulset-name> and verify each PVC maps to the correct pod.

Implementation Patterns

Pattern 1: Complete StatefulSet with Headless Service

A production-grade StatefulSet for a database workload with headless service, ordered pod management, and persistent storage.

---
apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  namespace: database
  labels:
    app: postgres
spec:
  ports:
    - port: 5432
      targetPort: 5432
      name: postgres
  clusterIP: None
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: database
  labels:
    app: postgres
    tier: database
spec:
  serviceName: postgres-headless
  replicas: 3
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: postgres
      tier: database
  template:
    metadata:
      labels:
        app: postgres
        tier: database
    spec:
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        fsGroup: 999
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
              name: postgres
          env:
            - name: POSTGRES_DB
              value: appdb
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              cpu: "1"
              memory: 2Gi
          livenessProbe:
            exec:
              command:
                - pg_isready
                - -U
                - "$(POSTGRES_USER)"
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            exec:
              command:
                - pg_isready
                - -U
                - "$(POSTGRES_USER)"
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: premium-ssd
        resources:
          requests:
            storage: 50Gi

Pattern 2: Stable DNS Naming (BAD vs GOOD)

Understanding how StatefulSet DNS names differ from Deployment-style naming.

# ❌ BAD: Service with clusterIP — pods get random names and load-balanced IPs
# No stable DNS entry per pod. Clients cannot address individual members.
apiVersion: v1
kind: Service
metadata:
  name: postgres-service
spec:
  type: ClusterIP  # ← This is wrong for StatefulSet
  selector:
    app: postgres

# ✅ GOOD: Headless service — each pod gets a stable DNS name
apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None  # ← Headless: no load balancing, one A record per pod
  selector:
    app: postgres

# Result: Pod postgres-0 resolves to:
#   postgres-0.postgres-headless.database.svc.cluster.local
# Pod postgres-1 resolves to:
#   postgres-1.postgres-headless.database.svc.cluster.local
# This stable DNS is the core value of StatefulSet over Deployment.

Pattern 3: Ordered Scaling and Partition Update

Controlled rollout and scaling for StatefulSets where order matters.

# Ordered scaling: pods are created in order 0, 1, 2... and
# terminated in reverse order N, N-1, ..., 0
# This ensures leaders are demoted before followers are removed.

# Progressive update: only update pods with ordinal >= partition
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 1
  # Effect: pod-0 stays on old revision; pods 1,2,... get updated.
  # This allows manual verification of pod-0 before proceeding.

# To update pod-0 as well:
# kubectl patch statefulset postgres --type='json' \
#   -p='[{"op": "replace", "path": "/spec/updateStrategy/rollingUpdate/partition", "value": 0}]'

# Manual update strategy — only update when pod is deleted:
spec:
  updateStrategy:
    type: OnDelete
  # Use when each member requires manual data migration before update.

# Delete pods in reverse order for safe shutdown:
# kubectl delete pod postgres-2
# kubectl delete pod postgres-1
# kubectl delete pod postgres-0
# PVCs are retained automatically after pod deletion.

def get_statefulset_pod_name(statefulset_name: str, ordinal: int) -> str:
    """Return the stable DNS name of a StatefulSet pod by ordinal.

    StatefulSet pods are named <statefulset-name>-<ordinal>, which provides
    deterministic identification for ordered operations.

    Args:
        statefulset_name: The name of the StatefulSet.
        ordinal: The zero-based ordinal of the pod.

    Returns:
        Stable pod name string.
    """
    if ordinal < 0:
        raise ValueError(f"Ordinal must be >= 0, got {ordinal}")
    return f"{statefulset_name}-{ordinal}"


def get_statefulset_dns_fqdn(
    pod_name: str,
    service_name: str,
    namespace: str
) -> str:
    """Construct the FQDN for a StatefulSet pod in the cluster DNS.

    Args:
        pod_name: The pod name (e.g., 'postgres-0').
        service_name: The headless service name.
        namespace: The Kubernetes namespace.

    Returns:
        Fully qualified DNS name for the pod.
    """
    if not pod_name or not service_name or not namespace:
        raise ValueError("pod_name, service_name, and namespace are required")
    return f"{pod_name}.{service_name}.{namespace}.svc.cluster.local"

Constraints

MUST DO

• Always create a headless service (clusterIP: None) before deploying the StatefulSet
• Set serviceName in the StatefulSet spec to the headless service name — this enables stable DNS
• Define volumeClaimTemplates with explicit accessModes, storageClassName, and storage request
• Use podManagementPolicy: OrderedReady (default) when startup/shutdown order matters
• Set revisionHistoryLimit to at least 3 to preserve rollout history
• Configure terminationGracePeriodSeconds: 60 for databases to allow flush and sync
• Use exec probes for databases (pg_isready, mysqladmin ping) rather than HTTP probes
• Match the StatefulSet's spec.selector.matchLabels exactly to spec.template.metadata.labels

MUST NOT DO

• Never use a regular ClusterIP service instead of a headless service — pods lose stable DNS
• Never mix StatefulSet with strategy.type: Recreate — it defeats ordered scaling
• Never set volumeClaimTemplates with accessModes: ReadWriteMany for single-writer databases
• Never scale a StatefulSet down and then back up expecting data to reappear — deleted pods' PVCs are retained
• Never set replicas: 0 on a production StatefulSet without first draining data
• Never use image: latest tag on StatefulSet pods — data-corrupting image changes during rollouts are irreversible

Output Template

When implementing a Kubernetes StatefulSet, produce the following:

1. Headless Service YAML — Service with clusterIP: None, proper port definitions, and matching selector labels.
2. StatefulSet YAML — Complete apps/v1 StatefulSet with serviceName, volumeClaimTemplates, updateStrategy, and pod template.
3. DNS Resolution Guide — Document the stable DNS names each pod will resolve to and how clients should connect.
4. Scaling Procedure — Step-by-step commands for scaling up and down in the correct order, with verification steps.

Related Skills

Live References

Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content.

• Kubernetes StatefulSets Documentation — Official guide to StatefulSet concepts, use cases, and lifecycle
• Headless Services for StatefulSets — How headless services provide stable DNS for StatefulSet pods
• Volume Claim Templates — Persistent volume provisioning per pod ordinal
• Pod Management Policies — OrderedReady vs Parallel scaling behavior
• Update Strategies — RollingUpdate with partition and OnDelete strategies
• Kubernetes API Reference — apps/v1 StatefulSet — Complete API schema for StatefulSet resources
• Database Deployment on Kubernetes — Best practices for running replicated stateful applications