Stop a runaway loop amplifier
A retry that spawns a retry, a webhook that triggers a callback that triggers the webhook, an agent that re-plans itself: these are loops where volume multiplies at each hop. They fill queues, wallets and rate limits in seconds. Break the cycle with a hop budget and a dedupe on the causal chain, not just a timeout.
Procedure
Trace one failing request end to end and count the calls it spawns. If a single request can generate more than a bounded number of downstream calls, you have an amplifier regardless of what the trace looks like on a happy path.
Propagate a hop count on every cross-service call and refuse beyond a small limit. Stick it in the same header as correlation so it travels automatically:
x-hop-count: 0 → 1 → 2 → 3 (drop)
if int(r.Header.Get("x-hop-count")) >= 3 { return 429, "hop budget exceeded" }
Dedupe on the causal chain: carry the originating event id and drop any work whose (origin_id, step) has already been processed. A webhook that fires in response to an event its own handler emitted is caught here:
if redis.set(f"seen:{origin}:{step}", 1, nx=True, ex=300) is None { return } // already handled
Make triggers edge-triggered on change, not level-triggered on state. A poller that enqueues work every time it sees a non-empty queue re-enqueues its own output forever; enqueue only when the value transitions.
Bound each queue's max depth and each consumer's in-flight, and shed when full rather than buffering without limit. An unbounded buffer hides the loop until memory runs out, which is a worse failure than a shed.
Put a global rate fuse on outbound calls per origin id: if one causal chain exceeds N calls/second, trip a breaker for that origin and alert. This catches loops you did not foresee.
Break the cycle at the design level where you can: make one direction of the loop asynchronous-only, or have the handler ignore events it caused (by emitter == self). Cheaper than any runtime guard.
Pitfalls
- A timeout that stops waiting but not the loop: the caller gives up while the callee keeps spawning.
- Correlation id forwarded but not the hop count, so the trace is observable but the loop is not bounded.
- Idempotency keyed on
(message_id) where the loop generates a fresh id at each hop, making every iteration look new.
- A retry with backoff but no cap on attempts, so a slow loop still multiplies, just over a longer window.
Verification
# emit one event and count the resulting calls; it must stay bounded
curl -s -X POST localhost:8080/events -d '{"type":"order.created","id":"e1"}'
curl -s 'localhost:9090/api/v1/query?query=sum(increase(calls_total[1m]))' | jq '.data.result[].value'
# expect a small bounded number, not growth; x-hop-count capped in logs
grep 'hop budget exceeded' /var/log/*.log | wc -l
redis-cli keys 'seen:e1:*' # dedupe keys present for each step
Report: the hop limit, the dedupe key scheme, the per-origin rate fuse, and an injected loop where volume stayed bounded and the breaker tripped instead of multiplying.
1---2name: stop-a-runaway-loop-amplifier3description: Use when two or more services or agents call each other and a failure multiplies instead of settling — adds a hop budget, correlation dedupe and a loop breaker so one request cannot fan out into thousands.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/stop-a-runaway-loop-amplifier6---78# Stop a runaway loop amplifier910A retry that spawns a retry, a webhook that triggers a callback that triggers the webhook, an agent that re-plans itself: these are loops where volume multiplies at each hop. They fill queues, wallets and rate limits in seconds. Break the cycle with a hop budget and a dedupe on the causal chain, not just a timeout.1112## Procedure13141. Trace *one* failing request end to end and count the calls it spawns. If a single request can generate more than a bounded number of downstream calls, you have an amplifier regardless of what the trace looks like on a happy path.15162. Propagate a hop count on every cross-service call and refuse beyond a small limit. Stick it in the same header as correlation so it travels automatically:17 x-hop-count: 0 → 1 → 2 → 3 (drop)18 if int(r.Header.Get("x-hop-count")) >= 3 { return 429, "hop budget exceeded" }19203. Dedupe on the causal chain: carry the originating event id and drop any work whose `(origin_id, step)` has already been processed. A webhook that fires in response to an event its own handler emitted is caught here:21 if redis.set(f"seen:{origin}:{step}", 1, nx=True, ex=300) is None { return } // already handled22234. Make triggers *edge-triggered on change*, not level-triggered on state. A poller that enqueues work every time it sees a non-empty queue re-enqueues its own output forever; enqueue only when the value *transitions*.24255. Bound each queue's max depth and each consumer's in-flight, and shed when full rather than buffering without limit. An unbounded buffer hides the loop until memory runs out, which is a worse failure than a shed.26276. Put a global rate fuse on outbound calls per origin id: if one causal chain exceeds N calls/second, trip a breaker for that origin and alert. This catches loops you did not foresee.28297. Break the cycle at the design level where you can: make one direction of the loop asynchronous-only, or have the handler ignore events it caused (by `emitter == self`). Cheaper than any runtime guard.3031## Pitfalls3233- A timeout that stops *waiting* but not the loop: the caller gives up while the callee keeps spawning.34- Correlation id forwarded but not the hop count, so the trace is observable but the loop is not bounded.35- Idempotency keyed on `(message_id)` where the loop generates a fresh id at each hop, making every iteration look new.36- A retry with backoff but no cap on *attempts*, so a slow loop still multiplies, just over a longer window.3738## Verification3940 # emit one event and count the resulting calls; it must stay bounded41 curl -s -X POST localhost:8080/events -d '{"type":"order.created","id":"e1"}'42 curl -s 'localhost:9090/api/v1/query?query=sum(increase(calls_total[1m]))' | jq '.data.result[].value'43 # expect a small bounded number, not growth; x-hop-count capped in logs44 grep 'hop budget exceeded' /var/log/*.log | wc -l45 redis-cli keys 'seen:e1:*' # dedupe keys present for each step4647Report: the hop limit, the dedupe key scheme, the per-origin rate fuse, and an injected loop where volume stayed bounded and the breaker tripped instead of multiplying.