# Backend Data Robustness

> Backend and data robustness at real scale + long-running job resilience — the gap that "build an API" skills leave. Trigger whenever code queries or writes a database inside a loop, processes large volumes, runs long (batch, replay, migration, cron), or defines probes/healthchecks/timeouts/retries — and ALWAYS before a >1h run, a parallel batch, or a deploy under sustained load, even if the user never says "performance" or "robustness".

- Skill: `carl-bouvet/backend-data-robustness` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add carl-bouvet/backend-data-robustness`
- Raw SKILL.md: https://api.skillmd.com/api/skills/carl-bouvet/backend-data-robustness/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: CARL-Bouvet (https://skillmd.com/u/carl-bouvet)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/carl-bouvet/backend-data-robustness

---


# Backend & Data Robustness

A discipline for backend code and data pipelines: behave correctly **at real
scale** and **survive without a human** over time. Distilled from real
production postmortems — every rule here cost hours or days.

Scope: **data & jobs under load + long-running resilience** — the gap left by general
backend-patterns skills (HTTP/API, pooling, DI), security skills (auth, OWASP, secrets)
and devops/SRE skills (CI/CD, IaC, runbooks). Complementary to all three, replacing none.

## The principle above the rules

**Measure before concluding, and test at target scale before launching.** A
measurement costs <30 s. A wrong hypothesis, or a scale bug discovered in
production, costs hours to days — orders of magnitude apart. An elaborate app that
can't run at scale is worth nothing.

## 1. Data access at scale — trigger: a query/write inside a loop, large volume

- **Index-first.** Every repeated filter/sort MUST have a covering index. After EVERY `dropIndex` or mode/schema change, verify `db.coll.find(filter).explain("executionStats")` → require `IXSCAN`, zero `COLLSCAN`. *Why: an upsert on `{a,b}` with no index = COLLSCAN over a growing collection = O(n²). Real case: a batch mode dropped the collection's only index — a 90-day TTL — because the history it was replaying would have been erased by it. Correct in itself. But that TTL was also the only thing bounding the collection's size: the upsert filter had never been indexed, which cost nothing while the TTL kept the collection small. Unbounded, the same upsert became a COLLSCAN over a growing collection, and a month-long job never finished. Recreate the index at the start of the job that needs it, not at boot — anything that drops the database between runs takes the index with it.*
- **Group your writes.** >100 DB ops in a loop → accumulate then `bulkWrite(ops, {ordered:false})` (with a per-item fallback if the bulk fails). *Why: 43,200 sequential `updateOne` = 43,200 round-trips + locks; batched by 5,000 = 9 round-trips. Don't accumulate the whole run in one command — the command cap is 16 MB, and an unbounded accumulator is the leak of the last bullet in this section. Measured on the live loop: the writing service's CPU 18.6% → 1.25%, mongo's 17.8% → 7.68%.*
- **Iterate what exists, not the dense domain.** On sparse data (time-series), loop over the keys actually present (`distinct(timestamps)`), not the theoretical range 0→N. *Why: computing 2.6M seconds/series, nearly all empty, costs in proportion to the domain, not the data.*
- **Bound every in-memory buffer/Map/queue** (hard cap + eviction LRU/TTL/FIFO+alert). *Why: an unbounded structure is a guaranteed memory leak → OOM.*

## 2. MongoDB pitfalls — trigger: mongo config, TTL, client, cache

- **TTL only on a BSON `Date` field.** `Date.now()` is an Int64 → the TTLMonitor **silently** ignores it (no-op, no log). Pattern: a dedicated `expireAt: Date` + `expireAfterSeconds: 0`, separate from your business `timestamp`. *127 GB leaked in 3 weeks on this trap.*
- **Size the engine cache against the *container* memory cap, never the host** — leave ≥25-30% of the cap above the cache for connections, sessions, journal and the OS. *Why: the process reads host RAM, not the cgroup limit, so a default sized on a dev laptop follows you into a smaller container. Real case: 3.5 GB cache in a 4 GB container (inherited from an 8 GB laptop) → permanent eviction at 80% cache used, mongo CPU pinned at 111% with unchanged insert throughput. Re-sized to 4.5 GB cache under a 6 GB cap: CPU 111% → 18%, same 30 inserts/s. Alert on cache-used %, not on cache size.*
- **One singleton `MongoClient`** at boot; never `connect/close` per operation. Back `serverHeartbeatFailed` with a periodic application-level ping + reconnect (`safeMongoOp`) — the driver doesn't always surface silent breaks (a zombie client recurred across two sessions before the periodic ping was added).
- **Native TTL > an application `deleteMany` loop** (the delete takes a collection W-lock that stalls inserts) — *provided* the field type is correct.

## 3. Before ANY long/parallel run — trigger: batch, migration, replay >a few minutes

- **Test at TARGET scale, not minimal scale.** An O(n²) bug is invisible at 1 day, fatal at 1 month. Run at least a sample at the real order of magnitude before the full run. *A pipeline "validated end-to-end" on 1 day never finished on 1 month.*
- **Benchmark at target concurrency, never extrapolate from N=1.** `throughput×N` fails past saturation (×5-20 degradation, ×17 in one incident: 257 POST/s alone, 150 POST/s total across 10 units). Bound: `PARALLEL ≈ nproc / cpu_per_unit`, then subtract 20-30% margin for the shared services on the same box (database, control plane). *35h wasted on a naïve linear projection.*
- **Canary.** Launch 1 unit (low failure cost) before 100%, with an explicit abort criterion: the first unit must complete and be marked DONE within a stated time budget (18h in our case); if not, kill it and intervene by hand. Don't chain the rest until the canary passes.
- **Pre-flight.** The script refuses to start if `load_avg / nproc > 1` at T0.

## 4. Long-running job wrapper (>2h, unattended) — trigger: you're writing a batch script

Combine ALL of these (each comes from a real failure):
- `flock -n` on a lockfile (released even on SIGKILL/OOM → cron restart is safe). A `*/5` auto-restart cron is safe **only** with `flock -n`, never blocking `flock` (otherwise instances pile up as zombies — 28 of them, in our case).
- State checkpoint **in a persistent dir, not `/tmp/`** (volatile), append-only (`DONE/FAILED <unit> <date>`), idempotent resume via `grep DONE`.
- **Timeouts everywhere**, not just global: `curl --max-time`, `timeout` on each sub-command. One hung sub-command freezes the whole batch without tripping the global timeout.
- **Retries** (3× + sleep) on every network/I/O op — a transient 1-2s glitch must not lose a unit.
- **Heartbeat file every 5 min** (stat-able) + `trap` cleanup on EXIT.
- **Partial failure ≠ success**: make the failure signal explicit and machine-readable — an append-only `FAILED <unit>` line in the state file — and propagate a non-zero exit code from the orchestrating layer on any failed unit. *Why: a wrapper that always exits 0 leaves whatever supervises it unable to distinguish a failed run from a successful one; it can only count `DONE` lines. If the wrapper does exit 0, then the `FAILED` entry IS the signal and the supervisor must read it — decide which of the two it is, and write it down. Define the abort threshold (e.g. >5% failures) before the run: nobody decides that mid-incident.*
- **Validate the real output** after each unit — file count and average size (e.g. ≥30 files, ≥10 KB each) — not just "no exception". *Structure checks catch a truncated, empty or half-written run. They do NOT catch a well-formed file with the wrong columns: schema validation (column count, header assertion, row typing) is the layer above, and the one most often missing.*

## 5. Hot-path / batch parity — trigger: a `BATCH_MODE` / `if (batch)` flag

- **Don't fork the hot path.** If the live path optimizes (groups its writes, keeps its indexes), the batch path MUST inherit the same optimizations. Batch changes only the **source/cadence**, not the persistence mechanism — factor out the shared write path. *Real case: batch had diverged from live (index dropped, writes ungrouped) — that was the bug.*
- The batch flag exists to disable live TTL/purge/loops that would corrupt a replay — not to reimplement a degraded version of the hot path.

## 6. Resilience & probes — trigger: healthcheck, probe, outbound call, sustained load

- **Shallow healthcheck**: "the process is listening" = HEALTHY, nothing more. Report dependency state as **cached last-known status**, never as a live ping inside the probe — a ping couples your liveness to that dependency and turns a 500 ms blip into a SIGKILL → cascade (one probe killed eleven services in our case). Prefer `curl -f` on the loopback address over `wget --spider`, which is lax on 3xx.
- **HEALTHY immediately, heavy init in the background** (`queueMicrotask` fire-and-forget). Blocking init > `start_period + retries×interval` → SIGTERM before ready → self-worsening crash-loop.
- **Under batch load that saturates the event loop: disable liveness + readiness, keep only a `startupProbe` with generous grace** (e.g. 10-15 min). *Nuance/contradicts the general "always add probes" rule: under ~250× live load, a single-threaded service stops answering `/health` in time — 87-96 restarts in 10h per replica. The database pod took 137 in 11h for a different reason: an `exec` probe whose process cold-start exceeds the timeout under CPU pressure. RAM was fine and the exit code was 0 = not an OOM, a probe false-negative.* Decouple: liveness tests "the process is alive", not "its dependencies are up".
- **Never drop silently.** Backpressure (503 to the caller) over dropping; never a bare `catch {}` or a lone `console.error` in place of a persistent error record (stable id `err_{module}_{fn}_{ctx}`, no variable timestamp). Every loss must be measurable, visible, alerted. Size buffers for the worst tolerated outage.
- **Circuit breaker per target** (closed→open after N failures→half-open after cooldown) on outbound calls: isolate a dead dependency without blocking the others (5 attempts × 5s timeout + `[500,1k,5k,10k]` backoff = ~40s of stalled send path per failed message, and 40s of buffer growth behind it).
- **Never permanently give up on a critical dependency.** Exponential backoff + jitter, cap ~30s. A bounded retry series is fine **provided** exhausting it triggers a cooldown and a fresh series, never a terminal state — name the constant after the cooldown, not after the giving-up.
- **Prefer WebSocket over HTTP polling** for high-frequency feeds (polling scales in CPU with frequency: JSON deserialization dominated the profile — an estimated 60-80% of the service's CPU at 90 polls/s).
- **Stagger multi-entity loops** (`for x of LIST` hitting the DB): a small pause between items + a deterministic offset between concurrent crons (don't fire everything at XX:00).

## 7. Incident diagnosis — trigger: it's slow / it's crashing / "it's probably X"

In increasing order of cost, don't skip an axis:
- **`uptime` + `top -bn1` BEFORE anything.** Load average measures saturation (run-queue); ratio `load/nproc`: ≤1 OK, >2 contention, >5 catastrophe.
- **`kubectl top` under-reports severely on a single-node cluster with host processes** — measured ×3.3 in one incident: `kubectl top node` said 27%, host `top` said 96%, because the metrics pipeline saw neither the control plane nor the runtimes started outside it. Not a constant: on a multi-node cluster where all load sits in pods, it is accurate. Cross-check with host `top`/`/proc/stat` before concluding "there's headroom".
- **`/proc/stat` is the CPU source of truth** (everything else derives from it). Delta over 2 samples. `iowait`→disk, `user+sys` without steal→over-parallelization, `sys>>user`→too many syscalls, `softirq` on one core→pinned IRQ.
- **`%steal`** settles "the hypervisor is stealing my CPU" vs "my fault" in 3s: <1% look at yourself, 5-10% keep an eye on it, >10% escalate to the host. Statistically it's almost always you — ours read 0.0% and killed the noisy-neighbour hypothesis on the spot.
- **The victim isn't the culprit**: the service crash-looping is often suffering upstream pressure. Read global metrics before diving into its logs. *Ours crash-looped for weeks; the cause was two stages upstream.*
- **`Exit 137` is a SIGKILL, and its origin depends on your orchestrator.** In Kubernetes, read `.status.containerStatuses[].lastState.terminated.reason`: `OOMKilled` true = kernel OOM, false = an external SIGKILL (kubelet, probe). In Swarm there is no such field — exit 137 means the healthcheck declared the container unhealthy and it was killed. Either way, confirm before assuming "out of memory": we lost hours to that assumption.
- **A too-tight CPU cap** (CFS throttling) freezes the event loop in 100ms slices and **simulates** overload — measure `nr_throttled/nr_periods` before suspecting load (`cpu.stat` in the container's cgroup; cAdvisor's `container_cpu_cfs_throttled_periods_total` in k8s). *Capped at 1.0 CPU, one service was throttled 42% of periods → a 100ms-sliced event loop → a task stretched to 13s → `/health` past its timeout → SIGTERM → crash-loop. Read as "overload"; it was the opposite.*
- **Never reboot/scale/reset before diagnosis**: the quick fix masks the permanent one — scaling a crash-loop just gives you more crash-loops.

## 8. Process & discipline — trigger: runtime change, bug spotted, a fix that "buys time"

- **Any runtime change** (`docker service update`, `kubectl patch`, a hand-created index) is **immediately persisted to the manifest + committed** — otherwise the next deploy overwrites it (3 identical regressions on this one).
- **A bug deferred "for later" recurs within 1-2 weeks**: fix it now if <10 min.
- **Any fix that "buys time"** (e.g. "4-8 weeks before the TTL retrim") **creates an alert on the threshold it postpones** — otherwise the incident is guaranteed right at the deadline.
- **Env-driven feature flags** (no rebuild) to toggle risky behavior: a CPU crisis was resolved with 2 env vars, 0 application lines.
- **Check `git status`/`git log`** before reasoning about deployed state (mea-culpas from stale notes).
- **Observability > monitoring**: emit a rich event per unit of work (`{unit, duration_ms, host_cpu, load_avg, db_active_clients}`) → ad-hoc `GROUP BY` for unknown-unknowns. An append-only JSONL is enough for a small project.
- **An alert = there's a problem AND you must act**: per-key cooldown + grace + dedup by id + recovery message, or the channel becomes noise you mute. Send fire-and-forget (`void` + `AbortSignal.timeout`), never `await` in the hot path.

## Further reading

The rules above are enough to act. For the underlying theory (mechanisms,
canonical methods, references), see `references/further-reading.md`.