Backend Data Robustness

Backend & Data Robustness

A discipline for backend code and data pipelines: behave correctly at real scale and survive without a human over time. Distilled from real production postmortems — every rule here cost hours or days.

Scope: data & jobs under load + long-running resilience — the gap left by general backend-patterns skills (HTTP/API, pooling, DI), security skills (auth, OWASP, secrets) and devops/SRE skills (CI/CD, IaC, runbooks). Complementary to all three, replacing none.

The principle above the rules

Measure before concluding, and test at target scale before launching. A measurement costs <30 s. A wrong hypothesis, or a scale bug discovered in production, costs hours to days — orders of magnitude apart. An elaborate app that can't run at scale is worth nothing.

1. Data access at scale — trigger: a query/write inside a loop, large volume

• Index-first. Every repeated filter/sort MUST have a covering index. After EVERY dropIndex or mode/schema change, verify db.coll.find(filter).explain("executionStats") → require IXSCAN, zero COLLSCAN. Why: an upsert on {a,b} with no index = COLLSCAN over a growing collection = O(n²). Real case: a batch mode dropped the collection's only index — a 90-day TTL — because the history it was replaying would have been erased by it. Correct in itself. But that TTL was also the only thing bounding the collection's size: the upsert filter had never been indexed, which cost nothing while the TTL kept the collection small. Unbounded, the same upsert became a COLLSCAN over a growing collection, and a month-long job never finished. Recreate the index at the start of the job that needs it, not at boot — anything that drops the database between runs takes the index with it.
• Group your writes. >100 DB ops in a loop → accumulate then bulkWrite(ops, {ordered:false}) (with a per-item fallback if the bulk fails). Why: 43,200 sequential updateOne = 43,200 round-trips + locks; batched by 5,000 = 9 round-trips. Don't accumulate the whole run in one command — the command cap is 16 MB, and an unbounded accumulator is the leak of the last bullet in this section. Measured on the live loop: the writing service's CPU 18.6% → 1.25%, mongo's 17.8% → 7.68%.
• Iterate what exists, not the dense domain. On sparse data (time-series), loop over the keys actually present (distinct(timestamps)), not the theoretical range 0→N. Why: computing 2.6M seconds/series, nearly all empty, costs in proportion to the domain, not the data.
• Bound every in-memory buffer/Map/queue (hard cap + eviction LRU/TTL/FIFO+alert). Why: an unbounded structure is a guaranteed memory leak → OOM.

2. MongoDB pitfalls — trigger: mongo config, TTL, client, cache

• TTL only on a BSON Date field. Date.now() is an Int64 → the TTLMonitor silently ignores it (no-op, no log). Pattern: a dedicated expireAt: Date + expireAfterSeconds: 0, separate from your business timestamp. 127 GB leaked in 3 weeks on this trap.
• Size the engine cache against the container memory cap, never the host — leave ≥25-30% of the cap above the cache for connections, sessions, journal and the OS. Why: the process reads host RAM, not the cgroup limit, so a default sized on a dev laptop follows you into a smaller container. Real case: 3.5 GB cache in a 4 GB container (inherited from an 8 GB laptop) → permanent eviction at 80% cache used, mongo CPU pinned at 111% with unchanged insert throughput. Re-sized to 4.5 GB cache under a 6 GB cap: CPU 111% → 18%, same 30 inserts/s. Alert on cache-used %, not on cache size.
• One singleton MongoClient at boot; never connect/close per operation. Back serverHeartbeatFailed with a periodic application-level ping + reconnect (safeMongoOp) — the driver doesn't always surface silent breaks (a zombie client recurred across two sessions before the periodic ping was added).
• Native TTL > an application deleteMany loop (the delete takes a collection W-lock that stalls inserts) — provided the field type is correct.

3. Before ANY long/parallel run — trigger: batch, migration, replay >a few minutes

• Test at TARGET scale, not minimal scale. An O(n²) bug is invisible at 1 day, fatal at 1 month. Run at least a sample at the real order of magnitude before the full run. A pipeline "validated end-to-end" on 1 day never finished on 1 month.
• Benchmark at target concurrency, never extrapolate from N=1. throughput×N fails past saturation (×5-20 degradation, ×17 in one incident: 257 POST/s alone, 150 POST/s total across 10 units). Bound: PARALLEL ≈ nproc / cpu_per_unit, then subtract 20-30% margin for the shared services on the same box (database, control plane). 35h wasted on a naïve linear projection.
• Canary. Launch 1 unit (low failure cost) before 100%, with an explicit abort criterion: the first unit must complete and be marked DONE within a stated time budget (18h in our case); if not, kill it and intervene by hand. Don't chain the rest until the canary passes.
• Pre-flight. The script refuses to start if load_avg / nproc > 1 at T0.

4. Long-running job wrapper (>2h, unattended) — trigger: you're writing a batch script

Combine ALL of these (each comes from a real failure):

• flock -n on a lockfile (released even on SIGKILL/OOM → cron restart is safe). A */5 auto-restart cron is safe only with flock -n, never blocking flock (otherwise instances pile up as zombies — 28 of them, in our case).
• State checkpoint in a persistent dir, not /tmp/ (volatile), append-only (DONE/FAILED <unit> <date>), idempotent resume via grep DONE.
• Timeouts everywhere, not just global: curl --max-time, timeout on each sub-command. One hung sub-command freezes the whole batch without tripping the global timeout.
• Retries (3× + sleep) on every network/I/O op — a transient 1-2s glitch must not lose a unit.
• Heartbeat file every 5 min (stat-able) + trap cleanup on EXIT.
• Partial failure ≠ success: make the failure signal explicit and machine-readable — an append-only FAILED <unit> line in the state file — and propagate a non-zero exit code from the orchestrating layer on any failed unit. Why: a wrapper that always exits 0 leaves whatever supervises it unable to distinguish a failed run from a successful one; it can only count DONE lines. If the wrapper does exit 0, then the FAILED entry IS the signal and the supervisor must read it — decide which of the two it is, and write it down. Define the abort threshold (e.g. >5% failures) before the run: nobody decides that mid-incident.
• Validate the real output after each unit — file count and average size (e.g. ≥30 files, ≥10 KB each) — not just "no exception". Structure checks catch a truncated, empty or half-written run. They do NOT catch a well-formed file with the wrong columns: schema validation (column count, header assertion, row typing) is the layer above, and the one most often missing.

5. Hot-path / batch parity — trigger: a BATCH_MODE / if (batch) flag

• Don't fork the hot path. If the live path optimizes (groups its writes, keeps its indexes), the batch path MUST inherit the same optimizations. Batch changes only the source/cadence, not the persistence mechanism — factor out the shared write path. Real case: batch had diverged from live (index dropped, writes ungrouped) — that was the bug.
• The batch flag exists to disable live TTL/purge/loops that would corrupt a replay — not to reimplement a degraded version of the hot path.

6. Resilience & probes — trigger: healthcheck, probe, outbound call, sustained load

• Shallow healthcheck: "the process is listening" = HEALTHY, nothing more. Report dependency state as cached last-known status, never as a live ping inside the probe — a ping couples your liveness to that dependency and turns a 500 ms blip into a SIGKILL → cascade (one probe killed eleven services in our case). Prefer curl -f on the loopback address over wget --spider, which is lax on 3xx.
• HEALTHY immediately, heavy init in the background (queueMicrotask fire-and-forget). Blocking init > start_period + retries×interval → SIGTERM before ready → self-worsening crash-loop.
• Under batch load that saturates the event loop: disable liveness + readiness, keep only a startupProbe with generous grace (e.g. 10-15 min). Nuance/contradicts the general "always add probes" rule: under ~250× live load, a single-threaded service stops answering /health in time — 87-96 restarts in 10h per replica. The database pod took 137 in 11h for a different reason: an exec probe whose process cold-start exceeds the timeout under CPU pressure. RAM was fine and the exit code was 0 = not an OOM, a probe false-negative. Decouple: liveness tests "the process is alive", not "its dependencies are up".
• Never drop silently. Backpressure (503 to the caller) over dropping; never a bare catch {} or a lone console.error in place of a persistent error record (stable id err_{module}_{fn}_{ctx}, no variable timestamp). Every loss must be measurable, visible, alerted. Size buffers for the worst tolerated outage.
• Circuit breaker per target (closed→open after N failures→half-open after cooldown) on outbound calls: isolate a dead dependency without blocking the others (5 attempts × 5s timeout + [500,1k,5k,10k] backoff = ~40s of stalled send path per failed message, and 40s of buffer growth behind it).
• Never permanently give up on a critical dependency. Exponential backoff + jitter, cap ~30s. A bounded retry series is fine provided exhausting it triggers a cooldown and a fresh series, never a terminal state — name the constant after the cooldown, not after the giving-up.
• Prefer WebSocket over HTTP polling for high-frequency feeds (polling scales in CPU with frequency: JSON deserialization dominated the profile — an estimated 60-80% of the service's CPU at 90 polls/s).
• Stagger multi-entity loops (for x of LIST hitting the DB): a small pause between items + a deterministic offset between concurrent crons (don't fire everything at XX:00).

7. Incident diagnosis — trigger: it's slow / it's crashing / "it's probably X"

In increasing order of cost, don't skip an axis:

• uptime + top -bn1 BEFORE anything. Load average measures saturation (run-queue); ratio load/nproc: ≤1 OK, >2 contention, >5 catastrophe.
• kubectl top under-reports severely on a single-node cluster with host processes — measured ×3.3 in one incident: kubectl top node said 27%, host top said 96%, because the metrics pipeline saw neither the control plane nor the runtimes started outside it. Not a constant: on a multi-node cluster where all load sits in pods, it is accurate. Cross-check with host top//proc/stat before concluding "there's headroom".
• /proc/stat is the CPU source of truth (everything else derives from it). Delta over 2 samples. iowait→disk, user+sys without steal→over-parallelization, sys>>user→too many syscalls, softirq on one core→pinned IRQ.
• %steal settles "the hypervisor is stealing my CPU" vs "my fault" in 3s: <1% look at yourself, 5-10% keep an eye on it, >10% escalate to the host. Statistically it's almost always you — ours read 0.0% and killed the noisy-neighbour hypothesis on the spot.
• The victim isn't the culprit: the service crash-looping is often suffering upstream pressure. Read global metrics before diving into its logs. Ours crash-looped for weeks; the cause was two stages upstream.
• Exit 137 is a SIGKILL, and its origin depends on your orchestrator. In Kubernetes, read .status.containerStatuses[].lastState.terminated.reason: OOMKilled true = kernel OOM, false = an external SIGKILL (kubelet, probe). In Swarm there is no such field — exit 137 means the healthcheck declared the container unhealthy and it was killed. Either way, confirm before assuming "out of memory": we lost hours to that assumption.
• A too-tight CPU cap (CFS throttling) freezes the event loop in 100ms slices and simulates overload — measure nr_throttled/nr_periods before suspecting load (cpu.stat in the container's cgroup; cAdvisor's container_cpu_cfs_throttled_periods_total in k8s). Capped at 1.0 CPU, one service was throttled 42% of periods → a 100ms-sliced event loop → a task stretched to 13s → /health past its timeout → SIGTERM → crash-loop. Read as "overload"; it was the opposite.
• Never reboot/scale/reset before diagnosis: the quick fix masks the permanent one — scaling a crash-loop just gives you more crash-loops.

8. Process & discipline — trigger: runtime change, bug spotted, a fix that "buys time"

• Any runtime change (docker service update, kubectl patch, a hand-created index) is immediately persisted to the manifest + committed — otherwise the next deploy overwrites it (3 identical regressions on this one).
• A bug deferred "for later" recurs within 1-2 weeks: fix it now if <10 min.
• Any fix that "buys time" (e.g. "4-8 weeks before the TTL retrim") creates an alert on the threshold it postpones — otherwise the incident is guaranteed right at the deadline.
• Env-driven feature flags (no rebuild) to toggle risky behavior: a CPU crisis was resolved with 2 env vars, 0 application lines.
• Check git status/git log before reasoning about deployed state (mea-culpas from stale notes).
• Observability > monitoring: emit a rich event per unit of work ({unit, duration_ms, host_cpu, load_avg, db_active_clients}) → ad-hoc GROUP BY for unknown-unknowns. An append-only JSONL is enough for a small project.
• An alert = there's a problem AND you must act: per-key cooldown + grace + dedup by id + recovery message, or the channel becomes noise you mute. Send fire-and-forget (void + AbortSignal.timeout), never await in the hot path.

Further reading

The rules above are enough to act. For the underlying theory (mechanisms, canonical methods, references), see references/further-reading.md.