arjunprabhulal Workflow Automation
An operational workflow is rarely one action — it's a chain: detect a condition, decide, act, verify, notify. Most automation failures happen not because a single step was wrong, but because the chain assumed the happy path and had no plan for a step failing halfway through, running twice, or firing on a burst of duplicate events.
Treat a workflow as a state machine you can inspect mid-flight, not a script that either finishes or silently dies. The workflow's job is to reach a known-good end state, not to execute a fixed sequence of commands.
1. Trigger on events, not on polling when you can help it
Polling wastes cycles and adds latency proportional to the poll interval; it also invites a race where two pollers both see the same unhandled condition. An event-driven trigger — a webhook, a queue message, a Kubernetes watch — reacts immediately and, if built on an at-least-once delivery system, gives you an explicit signal to deduplicate against instead of an implicit one you have to infer from timing.
- Prefer push over pull — a webhook or queue beats a cron job polling an API for changes.
- Carry a unique event ID through the whole chain so downstream steps can deduplicate.
- Treat "at least once" delivery as the default assumption, not an edge case to handle later.
Done when: the workflow's trigger is the event itself, not a poll loop guessing at freshness.
2. Make every step idempotent before you make it automatic
An automated step will eventually run twice — a retried webhook, a redelivered queue message, an operator re-running by hand after an ambiguous failure. If "create the resource" isn't safe to run twice, the second run either errors confusingly or creates a duplicate. Idempotency turns retries from a hazard into a free safety net.
- Key every action on a stable identifier (an ID, a desired-state hash) so re-running converges instead of duplicating.
- Check current state before acting — "ensure X exists" instead of "create X."
- Make the check and the act atomic where the underlying system allows it, so two concurrent runs don't both pass the check and both act.
Done when: running any step twice in a row produces the same end state as running it once.
3. Give retries a limit and a backoff, and a place to stop
An unbounded retry loop on a permanently broken step just burns resources while paging no one; an unbounded retry loop that isn't idempotent actively worsens the state with every attempt. Retries need three things: a cap, exponential backoff so a struggling downstream system gets relief instead of a hammering, and a terminal failure path that surfaces to a human instead of looping forever.
- Cap retry count or duration — after N attempts, stop and escalate.
- Back off between attempts — a fixed-interval retry against an overloaded dependency is a self-inflicted denial of service.
- Land failed runs somewhere visible — a dead-letter queue, a failed-runs dashboard — not a log line nobody reads.
Done when: a permanently failing step stops retrying on its own and produces a visible, actionable failure instead of a silent infinite loop.
4. Decide up front which steps a human must approve
Not every action belongs fully automated. The dividing line isn't "how often does this run," it's blast radius and reversibility — an action that's hard to undo or affects many users at once deserves a human checkpoint even if the automation to skip that checkpoint is easy to build.
- Auto-run what's cheap, reversible, and well-tested — restarting a crashed pod, rotating a log.
- Require approval for what's expensive to undo — deleting data, a production-wide config change, anything touching customer-visible state at scale.
- Make the approval step part of the workflow, not a side-channel Slack message the automation doesn't wait for.
See incident-response for how this same human-in-the-loop judgment applies under time pressure,
and runbooks for documenting the manual fallback when automation isn't trusted yet.
Done when: every workflow step is explicitly classified as auto-run or approval-required, and that classification is enforced by the workflow engine, not by convention.
5. Instrument the workflow, not just its steps
A workflow that logs each step's success but never records the whole chain's state leaves an operator reconstructing "where did this run get to" from scattered logs during an incident. Treat the workflow instance itself as a first-class object with a status.
- Emit a start and end event for the whole workflow run, correlated by the event ID from step 1.
- Expose current state — which step it's on, how many attempts, what it's waiting for — somewhere queryable, not just in logs.
- Alert on workflows stuck mid-chain the same way you'd alert on a failed one; a workflow that never finishes is often worse than one that fails fast.
Done when: an operator can answer "what is this workflow doing right now" without reading logs line by line.
Report
State which trigger type the workflow uses, whether every step is confirmed idempotent, the retry limit and backoff, and which steps require human approval.
Name the honest gap — usually a step that's "probably idempotent" but untested against a double-run, or a human-approval gate that exists in documentation but isn't enforced by the engine — rather than claiming the workflow is fully safe to retry blindly.