Restart Resilience

Restart Resilience

A restart is not a clean slate. A stateful worker that loses its place when the process dies — drops the task it was mid-way through, forgets its queue position, or skips scheduled work whose time passed while it was down — has a defect, not bad luck. Restarts happen for ordinary reasons: deploys, rescheduling, node pressure, crashes, out-of-memory. The bar is: after any restart, the worker resumes its full work from durable state.

This is the resume side of durability. The shutdown signal path (draining on SIGTERM within a grace period) is a separate, complementary concern — see graceful-shutdown-stateful-agents. This skill assumes the shutdown might be abrupt and asks: when the process comes back, does it pick up everything it owed?

What must survive a restart

• In-flight task state — the unit being processed and its progress, so it resumes from the last checkpoint rather than restarting from zero or losing it.
• Queue position — committed-but-unfinished items stay claimed/visible so a restart re-acquires them; they are not silently dropped or double-run.
• Scheduled work — jobs whose fire time elapsed during downtime are detected and run on recovery (catch-up), not skipped because "their moment passed".
• Session / continuity context — the working context needed to keep going; on a miss in the fast path, recover it from the durable store rather than starting fresh.

How to build it

1. Checkpoint in-flight state durably as work progresses — frequently enough that a restart loses at most a small, re-doable increment. A checkpoint that only exists in memory is not a checkpoint.
2. Recover from the store on startup, not just from the hot cache. When the in-memory/session lookup misses, fall back to reconstructing state from the durable record. A miss must trigger recovery, never a silent fresh start.
3. Make resume idempotent. A re-acquired or re-delivered task must not double-apply its effect. Use a stable work key + an applied/seen marker so replays are safe.
4. Keep queue claims durable and reclaimable. An in-progress claim that outlives the process (visibility timeout, lease, or persisted claim) lets a restart take the item back instead of orphaning it.
5. Reconcile scheduled work on boot. Compare the schedule against last-run markers and run anything whose window elapsed while down, exactly once.

Verify it

Test the brutal path, not the polite one. Kill the process mid-task (not a clean exit) and confirm on restart it: resumes the in-flight unit, re-acquires its queue position, runs the schedule it missed, and applies nothing twice. "It restarted and kept running" is not the test — "it restarted and lost nothing" is.

Anti-patterns

• progress held only in memory, so a crash restarts the unit from zero or drops it
• treating a session/context cache miss as "new start" instead of recovering from the store
• scheduled jobs that silently skip any window the worker was down for
• non-idempotent resume that double-applies a re-delivered task
• queue items that vanish when the worker that claimed them dies (no lease, no visibility timeout, no persisted claim)
• declaring resilience proven by a clean restart only, never testing a kill mid-task

Done means

After an abrupt mid-task kill, the worker recovers from durable state: it resumes in-flight work, re-acquires its queue position, runs scheduled work it missed, and applies each unit exactly once — proven by a kill-mid-task recovery test, not a clean-restart test.