Kubernetes Service Lifecycle

Kubernetes Service Lifecycle

Purpose

Make a Java service correct at the two moments the orchestrator controls: when it is declared ready, and when it is told to stop. Deploy-time 502s can come from lifecycle, routing, resource pressure or application failures. Correlate request failures with pod, probe, endpoint and shutdown events before attributing the cause.

The failure this prevents is the probe that answers the wrong question. A liveness probe that checks a downstream dependency converts a partial degradation into a total outage: the database wobbles, every replica fails liveness, the kubelet restarts all of them at once, and now nothing is serving even after the database recovers.

Compatibility and evidence

Inspect the deployed JDK, resolved Boot/Framework/Kafka versions, image entrypoint and signal forwarding, cluster version/feature gates, and effective Deployment/Service settings. The references use partial Java 17-compatible sketches; virtual-thread APIs require Java 21. Do not upgrade the project to apply them. Missing runtime evidence permits a conditional configuration finding, not a confirmed incident diagnosis.

Workflow

1. Assign each probe its own question. Liveness = "restart me, I am unrecoverable in process". Readiness = "send me traffic now". Startup = "I am still booting, do not judge me yet". Distinct semantics need not mean three endpoints: startup may reuse liveness with a different budget. Verify what each check actually observes.
2. Strip dependencies out of liveness. Liveness must depend on nothing outside the process. If restarting the process cannot fix the condition, it does not belong in liveness.
3. Do bounded probe arithmetic. Detection includes initial delay, probe scheduling, execution/timeout and consecutive thresholds; readiness may run more often while unready. Treat period × threshold as an approximation, write best/worst expectations, and test under throttling and pauses. See references/probe-and-shutdown-configuration.md.
4. Replace guessed startup delays with a startup probe. startupProbe (GA since Kubernetes 1.20) suspends liveness and readiness until it first succeeds, so a slow boot gets a long budget without making crash detection slow forever.
5. Budget the shutdown as a sum. terminationGracePeriodSeconds must exceed preStop plus the application's own drain, with margin. It is one countdown, not one per stage; overrun means SIGKILL mid-request.
6. Enumerate the in-flight work that is not an HTTP request — Kafka consumers, @Scheduled jobs, executors, queue leases — and give each an explicit stop. See references/draining-non-http-work.md. Then check the disruption path: a minAvailable: 1 budget with one healthy replica blocks compliant eviction; allowing eviction instead can create an availability gap until a replacement is ready.

Probe decision block

Use a liveness probe when:
- the process has a reachable state that only a restart clears — a deadlock, an
  exhausted internal thread pool, a wedged event loop — and you can detect it in process.
Avoid a liveness probe when:
- the check touches a database, a cache, a broker or another service. A shared dependency
  makes every replica fail simultaneously, which is a correlated failure you built.
- you cannot name the in-process condition it detects. Then it has no signal, only risk;
  omitting liveness entirely is a legitimate configuration.
Use a readiness probe when:
- the pod can be temporarily unable to serve while still being worth keeping — warming a
  cache, reconnecting, or shedding under local overload.
Avoid putting a hard dependency in readiness when:
- the whole fleet shares it. All replicas leave the endpoint list at once and the Service
  has no backends, which is worse than serving degraded responses.
Prefer a startup probe instead when:
- boot time varies with data volume, cluster load or CPU throttling, i.e. whenever you
  would otherwise have guessed initialDelaySeconds.

Rules

• Readiness failure makes the pod unready and excludes it from normal ready-endpoint routing; it does not itself restart the container. Check publishNotReadyAddresses, custom consumers and existing connections before assuming traffic stops. Liveness failure at its threshold triggers container restart handling. Choosing the wrong probe turns a routing decision into a restart storm.
• The probe endpoint must do no business work and have no side effect. It runs on every pod every periodSeconds forever: a query inside it is permanent background load, and a write inside it is a bug the kubelet triggers on a schedule.
• timeoutSeconds is part of the failure-detection budget. Derive it from a lightweight local check's measured tail plus jitter, then decide how many consecutive misses justify action; "greater than worst case" is unusable when the worst case is unbounded. successThreshold must be 1 for liveness and startup probes.
• Local termination and data-plane convergence are concurrent. Terminating EndpointSlice endpoints normally become not ready (check publishNotReadyAddresses), but proxies, ingresses, clients and persistent connections converge on their own timelines. A measured preStop sleep can bridge legacy data planes; explicit readiness refusal, connection draining and load-balancer behavior are preferable when supported. Sleeping is a workaround, not a universal protocol.
• The native sleep lifecycle handler is version-dependent. On clusters without it, preStop.exec needs a real binary in the image; distroless/scratch images often lack one. A failed hook is observable through pod events (FailedPreStopHook) but termination continues, so alerting must not rely on application logs.
• preStop runs inside terminationGracePeriodSeconds, not before it. A 30 s grace period with a 20 s preStop leaves the application 10 s, then SIGKILL.
• Spring Boot enables graceful web shutdown by default from 3.4; earlier supported lines require server.shutdown=graceful. Pin the service's Boot version and verify effective behavior; the window is governed by spring.lifecycle.timeout-per-shutdown-phase, which is not a total shutdown deadline.
• A container killed after grace expiry and one killed for memory can both surface as 137. Correlate terminated reason/signal, events, cgroup counters and timestamps; OOMKilled is strong orchestrator evidence, not the only possible record. Heap sizing belongs to container-awareness.
• CPU quota often affects JVM startup because class loading, verification and compilation create bursts, but "hardest" is workload-dependent. Measure cold-start distribution in the same quota and node conditions used in production.
• A rolling update runs two versions concurrently by design. The API contract consequence is rpc-and-api-contracts; the data consequence is yours — a schema change must be readable by both versions at once.
• A PodDisruptionBudget constrains only disruptions routed through the Eviction API. It does not prevent a node crash or direct delete, and workload controllers are not constrained by it during rollout. minAvailable: 1 with one healthy replica blocks compliant eviction; operators can still bypass it or time out, so call it unavailable by policy, not immortal.
• Never claim a rolling update is zero-downtime because the manifest has a readiness probe. Validate the stated SLO with an open-loop client through repeated deploys: record offered and completed requests, timeouts, resets, unexpected status codes and latency. Zero errors in a finite run is evidence for those conditions, not a universal guarantee.

Output

Return the observed failure timeline or configuration risk, the smallest justified change, the total shutdown budget including sequential phases, and a validation with explicit pass criteria. Separate executed checks from rollout or fault tests still needed.

References

• Probe and shutdown configuration — a correct three-probe manifest fragment with the timing arithmetic derived, the preStop/grace-period/drain budget as one sum, and the Spring Boot properties and Actuator health groups behind it. Read when writing or reviewing a Deployment, or when a probe setting is being changed.
• Draining work that is not an HTTP request — Kafka consumers, @Scheduled, executor shutdown, Spring's stop ordering, and a concrete test that proves a shutdown actually drains. Read when the service consumes a queue, runs scheduled work, or owns its own threads.