rockclaver Workflow Engine Design
Build the engine layer — not individual idempotent operations (see idempotent-financial-workflows), but the runtime that sequences them durably.
Orchestration vs Choreography
- Orchestration: central coordinator holds explicit state; easier to observe, debug, and audit. Use for multi-step money flows and compliance-sensitive sequences.
- Choreography: services react to events on a shared bus; decoupled, harder to trace. Use for loose fan-out notifications.
- Hybrid: orchestrator per bounded context, events between contexts.
State Machine Design
- Name every state and every valid transition before writing code.
- Store state durably before executing side effects — never derive current state from side-effect logs.
- Terminal states:
completed,failed,cancelled,compensated. - Persist the full input/output of each step; a replay must reproduce the same decisions without re-running external calls.
Durable Execution Primitives
- Step isolation: each step reads input, executes idempotently, writes output + new state atomically.
- Outbox: cross-service steps commit local state + outbox message in one transaction; a relay delivers externally.
- Timers/delays: store as a scheduled-event record; a polling worker or dedicated scheduler fires it. Never
sleep(). - Suspension/resume: workflow pauses waiting for an external callback; store a resume token; callback delivers it and the engine re-schedules the workflow.
Compensating Transactions (Sagas)
- For each step with an external side effect, register a compensating action before execution.
- On failure, execute compensations in reverse order — each compensation must also be idempotent.
- Record compensation outcome; never silently swallow compensation errors.
Worker Concurrency Model
- Lease-based polling: worker claims a task with a TTL lease; heartbeats extend it; expiry allows re-claim.
- Fan-out/fan-in: parallel branches share the workflow ID with a branch index; fan-in waits for all branch completions.
- Backpressure: bounded worker pool; queue-depth metric drives horizontal scaling.
Workflow Versioning
- Assign a schema version to each workflow definition. Running instances finish on the version they started.
- Safe changes: add optional steps, add new terminal paths.
- Breaking changes (require a new version): remove, reorder, or rename steps.
- Migration: drain old version to completion, then decommission the handler.
Dead-Letter and Stuck Workflows
- Max attempt limit per step → move to
dead_letteredstate with full context snapshot. - Ops tooling required: list stuck/dead-lettered workflows; replay from a specific step; force-complete a terminal step.
Testing Long-Running Workflows
- Unit: synchronous in-process runner — no queues, no real timers. Inject step fakes for each external outcome (success, transient failure, permanent failure, duplicate callback).
- Time advance: trigger timer steps programmatically; never
sleep()in tests. - Regression: replay a recorded production workflow trace and assert final state.