Kubernetes Graceful Shutdown

Kubernetes Graceful Shutdown

TL;DR: A pod receiving SIGTERM is not the same moment that traffic stops arriving. The kubelet and the Endpoints/Service controller run concurrently, so kube-proxy on every node may take seconds to remove your pod from its iptables rules. The fix is a preStop hook that sleeps for ~5–15 seconds before SIGTERM, giving the data plane time to converge, plus a signal handler in your app that drains in-flight work within terminationGracePeriodSeconds.

Jump to your fire

Decision diagram

flowchart TD
  A[Pod about to be deleted<br/>rolling deploy, scale-in, eviction] --> B[API server sets deletionTimestamp]
  B --> C[/Concurrent: kubelet + EndpointSlice controller/]
  C --> D[Kubelet runs preStop hook<br/>blocking, before SIGTERM]
  C --> E[EndpointSlice marks endpoint terminating]
  D --> F{preStop done?}
  E --> G[kube-proxy on each node updates iptables<br/>+ external LB deregisters target]
  F -->|Yes| H[Kubelet sends SIGTERM to PID 1]
  H --> I{App drains in-flight requests<br/>within terminationGracePeriodSeconds?}
  I -->|Yes, exit 0| J[Pod removed cleanly]
  I -->|No, deadline exceeded| K[Kubelet sends SIGKILL]
  K --> L[In-flight requests die mid-response]

The hazard is steps E + G racing against D + H. If your app stops accepting connections before G completes, kube-proxy still routes traffic to your pod's IP — connection refused.

1. The endpoint removal race

From the Kubernetes EndpointSlices docs:

The terminating condition indicates that the endpoint is terminating. For endpoints backed by a Pod, this condition is set when the Pod is first deleted (that is, when it receives a deletion timestamp, but most likely before the Pod's containers exit).

And:

With kube-proxy running on each Node and watching EndpointSlices, every change to an EndpointSlice becomes relatively expensive since it will be transmitted to every Node in the cluster.

Translation: when the API server stamps your pod with deletionTimestamp, the EndpointSlice controller marks the endpoint terminating immediately — but the actual update to iptables on every node, plus deregistration from any external load balancer (AWS NLB, GCP LB), takes time. Tens of milliseconds in a small cluster, several seconds in a large one or with a slow LB.

In that gap, traffic still arrives at your pod's IP. If your app has already started shutting down its listener, those new connections fail.

2. The pod termination sequence

From the Container Lifecycle Hooks docs:

PreStop hooks are not executed asynchronously from the signal to stop the Container; the hook must complete its execution before the TERM signal can be sent.

If terminationGracePeriodSeconds is 60, and the hook takes 55 seconds to complete, and the Container takes 10 seconds to stop normally after receiving the signal, then the Container will be killed before it can stop normally, since terminationGracePeriodSeconds is less than the total time (55+10).

Ordered sequence (verified across the lifecycle and EndpointSlice docs):

terminationGracePeriodSeconds defaults to 30 seconds.

The grace-period budget is shared between preStop and the post-SIGTERM drain. A 60-second grace with a 30-second preStop sleep leaves only 30 seconds to drain in-flight work. Size the budget accordingly.

3. The preStop sleep recipe

The pattern: don't start draining until traffic has stopped arriving. Sleep gives the data plane time to converge while the app keeps serving normally.

spec:
  containers:
  - name: api
    image: my-api:latest
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 15"]
  terminationGracePeriodSeconds: 60   # must be > preStop sleep + drain time

Sleep duration rationale (operational rules of thumb — kubernetes.io documents the race but not the exact tuning):

Tune by measuring: enable access logs, do a rolling deploy, count requests that landed after SIGTERM. The right sleep is the smallest value that brings that count to zero.

Kubernetes 1.29+ adds a native sleep action so you don't need /bin/sh:

lifecycle:
  preStop:
    sleep:
      seconds: 15

The exec sleep form remains universally compatible with older clusters.

Don't forget the readiness probe

A complementary trick: have your app fail readiness before it stops accepting traffic. This makes EndpointSlice mark the pod NotReady faster on its own.

// Pseudocode
let ready = true
process.on('SIGTERM', () => { ready = false; /* drain... */ })
app.get('/readyz', (req, res) => res.status(ready ? 200 : 503).end())

But: this does not replace the preStop sleep. The readiness gate helps the next propagation cycle; the sleep covers the current in-flight traffic. Use both.

4. Node.js shutdown skeleton

From the Node.js HTTP docs: server.close() "stops the server from accepting new connections and closes all connections connected to this server which are not sending a request or waiting for a response." Since v19.0.0, it closes idle keep-alive connections automatically. closeIdleConnections() (v18.2.0) and closeAllConnections() (v18.2.0) are the explicit hammers.

import http from 'node:http'

const server = http.createServer((req, res) => { /* ... */ })
server.listen(8080)

let shuttingDown = false

function shutdown() {
  if (shuttingDown) return
  shuttingDown = true

  // Optional: flip readiness so endpoint is removed faster
  // (your /readyz handler reads `shuttingDown`)

  // Stop accepting new connections; finish in-flight requests
  server.close((err) => {
    process.exit(err ? 1 : 0)
  })

  // Kick idle keep-alive sockets so close() can resolve faster
  server.closeIdleConnections()  // Node 18.2+

  // Hard deadline shorter than terminationGracePeriodSeconds
  setTimeout(() => {
    server.closeAllConnections()  // Node 18.2+: forcibly drop active requests
    process.exit(1)
  }, 25_000).unref()
}

process.on('SIGTERM', shutdown)
process.on('SIGINT', shutdown)

Pitfall: when running Node under npm start or a shell wrapper, PID 1 is the shell, not Node — and the shell may not forward SIGTERM. Either run Node directly (CMD ["node", "server.js"] in the Dockerfile, no shell) or use a process supervisor like tini (--init in docker run, or set shareProcessNamespace: true carefully).

5. Go shutdown skeleton

From the net/http docs: Server.Shutdown "gracefully shuts down the server without interrupting any active connections... by first closing all open listeners, then closing all idle connections, and then waiting indefinitely for connections to return to idle and then shut down."

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{Addr: ":8080", Handler: mux()}

    // Canonical signal-to-context bridge (Go 1.16+)
    ctx, stop := signal.NotifyContext(context.Background(),
        syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    go func() {
        if err := srv.ListenAndServe(); err != nil &&
            !errors.Is(err, http.ErrServerClosed) {
            log.Fatalf("listen: %v", err)
        }
    }()

    <-ctx.Done()
    log.Println("shutdown signal received")

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("graceful shutdown failed: %v", err)
        _ = srv.Close()  // hard close — drop in-flight
    }
}

signal.NotifyContext is the idiomatic Go pattern; the returned context cancels on SIGTERM/SIGINT. Server.Close() is the immediate (non-graceful) variant; use it as the last-resort fallback.

Anti-patterns

Novice / Expert / Timeline

Timeline test: roll a fresh deploy under sustained synthetic traffic. An expert deployment shows zero 5xx in client logs. A novice deployment shows a brief blip of connection-refused / ECONNRESET errors per pod replaced.

Quality gates

A graceful-shutdown change ships when:

• Test: preStop hook exists with explicit sleep duration; terminationGracePeriodSeconds > preStop_sleep + drain_budget. Verified by a kubectl plugin or a CI lint of manifests.
• Test: Rolling deploy under synthetic traffic — measure post-SIGTERM request count from app access logs, confirm zero connection refused / connection reset errors at the client.
• Test: SIGTERM handler exists and is wired to PID 1 (no shell-wrapped CMD swallowing the signal). Verify with docker run --rm IMAGE pgrep -P 0 or kubectl exec.
• Test: Readiness probe flips to non-ready during shutdown drain (curl /readyz after triggering SIGTERM, expect 503).
• Test: Drain has a hard ceiling — even if requests are stuck, the process exits before terminationGracePeriodSeconds expires (shorter setTimeout / context.WithTimeout).
• Manual: Background workers (queue consumers, cron) also respond to the shutdown signal — not just the HTTP server.

NOT for this skill

• Database connection draining specifically (use postgres-connection-pooling for idle_in_transaction_session_timeout and pool-side concerns)
• Redis client shutdown (use redis-patterns-expert)
• Deployment / rollout strategies (use kubernetes-rollout-strategies or argo-rollouts-canary)
• Pod disruption budgets and node draining (use kubernetes-pdb-and-drains)
• Service mesh sidecar lifecycle (Istio, Linkerd) — sidecars introduce their own shutdown ordering issues; use service-mesh-shutdown-ordering

Sources

• Kubernetes: Pod Lifecycle
• Kubernetes: Container Lifecycle Hooks — preStop is synchronous-before-SIGTERM
• Kubernetes: EndpointSlices — terminating condition is set "before the Pod's containers exit"
• Kubernetes: Safely Drain a Node — kubectl drain, PDB interaction
• Node.js: http.Server.close() and friends — closeIdleConnections() and closeAllConnections() added in v18.2.0
• Go: net/http.Server.Shutdown
• Go: os/signal.NotifyContext