Coding Resilience

Design for self-healing; never assume an external resource is up

Overview

No external resource is guaranteed. A connection, remote service, API, proxy, DNS name, disk, CPU/memory, mount, or DB pool can be absent, slow, flaky, rate-limited, or vanish mid-run. Never treat one as granted or stable. Build the self-healing in from the start instead of coding the happy path and bolting on error handling later. Resilience lives at the I/O boundary (the adapter), not in the domain - see bitranox:coding-python-clean-architecture.

When to use

• Calling a network service, HTTP/REST API, proxy, or DNS name.
• Reading/writing a DB, connection pool, mount, or remote filesystem.
• Driving a remote host over SSH (bitranox:compuse-ssh) or VNC (bitranox:compuse-vnc).
• Bulk-fetching from a host that throttles per IP (see bitranox:net-rotating-proxies).
• A heavy step that needs disk / CPU / memory headroom that may not be there.
• Any adapter sitting at an I/O boundary.

Patterns

For the modern Python pick behind each, see bitranox:coding-python-use-modern-libraries.

• Retry with backoff + jitter, always under a hard timeout. Retry only transient failures (timeouts, connection resets, 429/503); never retry a deterministic 4xx. Exponential backoff with random jitter avoids a thundering herd. Cap the attempts AND put a timeout on every individual wait and on the whole operation - an unbounded retry with no timeout is a hang, not resilience. Python: tenacity (@retry(stop=stop_after_attempt(n), wait=wait_exponential_jitter())); set an explicit client timeout (httpx2.Client(timeout=...)) and asyncio.wait_for(coro, timeout=).
• Health-check, evict, replace. Probe a member before/while using it; drop a faulty, flaky, or slow one and pull in a fresh one. Track per-member success/failure and evict on a failure ratio, not just on a hard-dead connection.
• Maintain a pool at a target size with margin. Keep ~100% spare so the fast healthy members carry the load and the rest are warm backup; rotate so no single member is hammered. Worked example: bitranox:net-rotating-proxies (right-sized, speed-weighted pick, evict-on-failure).
• Rediscover; do not trust a cached list. Re-resolve / re-validate endpoints each run - yesterday's live host may be dead now. A cached endpoint list is a stale assumption.
• Top up to the target in the background so attrition is invisible to the running work.
• Circuit breaker for a repeatedly-failing dependency. After N consecutive failures, OPEN the circuit: fail fast (or serve a fallback) instead of hammering a dead service; after a cool-off, try one probe (HALF-OPEN) and CLOSE on success. Stops one sick dependency cascading into a stall.
• Degrade gracefully. On partial failure return a partial result + a clear warning + a non-zero exit - never a silent wrong answer. If a target resolves somewhere unexpected (e.g. a public host to an internal IP), say so.
• Guard resources. Cap concurrency, memory, and payload/collection size (unbounded input is a DoS vector - see bitranox:coding-input-sanitization). Check disk / CPU / memory headroom before a heavy step (shutil.disk_usage, os.cpu_count, load) rather than assuming it is there; stream or paginate large data instead of materializing it whole.
• ISOLATE each item in a batch of INDEPENDENT tasks. A loop over N unrelated units (sync 38 apps, poll 20 hosts) must not let one failure end the run. set -euo pipefail around such a loop does exactly that: the FIRST failing unit aborts the script, every later unit silently never runs, and the log shows no error because nothing errored - it stopped. The symptom is "sometimes only some of them updated", which reads like a flaky dependency and is not. Catch per unit, record the failure with the unit's name, keep going, and exit non-zero at the END with a tally. Reserve fail-fast for a genuine DEPENDENCY chain, where later steps are meaningless once one fails.

Common mistakes