# Workflow Engine Design

> Design and build durable workflow engines — state machines, saga orchestration, durable execution, timers, versioning, and worker concurrency. Use when building a workflow runtime, orchestrator, saga engine, or long-running job system.

- Skill: `rockclaver/workflow-engine-design` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rockclaver/workflow-engine-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rockclaver/workflow-engine-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: rockclaver (https://skillmd.com/u/rockclaver)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rockclaver/workflow-engine-design

---


# Workflow Engine Design

Build the engine layer — not individual idempotent operations (see `idempotent-financial-workflows`), but the runtime that sequences them durably.

## Orchestration vs Choreography

- **Orchestration**: central coordinator holds explicit state; easier to observe, debug, and audit. Use for multi-step money flows and compliance-sensitive sequences.
- **Choreography**: services react to events on a shared bus; decoupled, harder to trace. Use for loose fan-out notifications.
- **Hybrid**: orchestrator per bounded context, events between contexts.

## State Machine Design

- Name every state and every valid transition before writing code.
- Store state durably **before** executing side effects — never derive current state from side-effect logs.
- Terminal states: `completed`, `failed`, `cancelled`, `compensated`.
- Persist the full input/output of each step; a replay must reproduce the same decisions without re-running external calls.

## Durable Execution Primitives

- **Step isolation**: each step reads input, executes idempotently, writes output + new state atomically.
- **Outbox**: cross-service steps commit local state + outbox message in one transaction; a relay delivers externally.
- **Timers/delays**: store as a scheduled-event record; a polling worker or dedicated scheduler fires it. Never `sleep()`.
- **Suspension/resume**: workflow pauses waiting for an external callback; store a resume token; callback delivers it and the engine re-schedules the workflow.

## Compensating Transactions (Sagas)

- For each step with an external side effect, register a compensating action before execution.
- On failure, execute compensations in reverse order — each compensation must also be idempotent.
- Record compensation outcome; never silently swallow compensation errors.

## Worker Concurrency Model

- **Lease-based polling**: worker claims a task with a TTL lease; heartbeats extend it; expiry allows re-claim.
- **Fan-out/fan-in**: parallel branches share the workflow ID with a branch index; fan-in waits for all branch completions.
- **Backpressure**: bounded worker pool; queue-depth metric drives horizontal scaling.

## Workflow Versioning

- Assign a schema version to each workflow definition. Running instances finish on the version they started.
- Safe changes: add optional steps, add new terminal paths.
- Breaking changes (require a new version): remove, reorder, or rename steps.
- Migration: drain old version to completion, then decommission the handler.

## Dead-Letter and Stuck Workflows

- Max attempt limit per step → move to `dead_lettered` state with full context snapshot.
- Ops tooling required: list stuck/dead-lettered workflows; replay from a specific step; force-complete a terminal step.

## Testing Long-Running Workflows

- **Unit**: synchronous in-process runner — no queues, no real timers. Inject step fakes for each external outcome (success, transient failure, permanent failure, duplicate callback).
- **Time advance**: trigger timer steps programmatically; never `sleep()` in tests.
- **Regression**: replay a recorded production workflow trace and assert final state.