Upgrade A Live Service Safely

Upgrade a Live Service Safely

Mission: upgrade a service without taking it down — and when something does go wrong, recover fast with a rollback you prepared before you needed it.

🔴 Load this skill and its scripts/ BEFORE writing any deploy tooling. On One run spent ~40 minutes hand-rolling staging commands, skipped the test-port smoke entirely, and was about to cut over to production before anyone checked whether a packaged procedure already existed. It did: the bundled scripts/stage_and_smoke.sh already contained the test-port instance, the DB-copy isolation, the live-MainPID invariant, and the real-inference smoke — none of which the hand-rolled version had. Reach for the packaged script before improvising one. Adapting a proven script is faster than writing a worse one, and the parts you would have skipped are exactly the parts that catch problems.

The governing rule: never let the build and the running service share a directory, a CPU budget, or an SSH session. Every serious outage in this class comes from violating one of those three.

The strongest version of that rule: don't build on the box at all. Build in CI (free arm64 runners on public repos), ship an artifact, unpack to a new release dir, smoke it on a test port, then swap. Proven end-to-end — see references/ci-offload-build-and-stage.md, with a copy-and-adapt templates/ci-standalone-build.yml and scripts/stage_and_smoke.sh. Filling that template in for a specific service — which facts to pull off the live unit first, and the two asserts that make the artifact trustworthy — is references/adapting-the-ci-build-template.md. Before sizing a box for a build, measure the ratio: on the host runtime RSS was 1.2 GB while the build peaked at 15 GB — 16 GB had been provisioned purely to survive a monthly build. 🔴 Adapt the packaged script by READING the unit, not guessing it. One systemctl --user show <unit> -p ExecStart -p WorkingDirectory -p EnvironmentFiles --value answers entry point, env file, and working directory. Three failed staging runs all came from skipping it: source-ing a file that systemd loads via EnvironmentFile= (that is NOT shell — unquoted parens in user-agent values are a bash syntax error but valid to systemd), running the entry relative to the release root instead of WorkingDirectory, and inferring artifact layout from a top-level ls when both releases carried both trees. Also covers why a source-level string usually CANNOT serve as the discriminating assert (the bundle is minified — 1000 became 1e3, numeric separators vanished, and the docstring compiled away entirely) and how to locate the compiled site via a log or SQL literal that survives minification: references/adapting-stage-and-smoke-to-a-real-unit.md. 🔴 Getting deploy commits off the box is its own failure class. A scratch clone made with git clone --shared <local checkout> has origin pointing at the LOCAL directory, so git push -q exits 0 having pushed nowhere and CI then fails with upload-pack: not our ref. Never -q a push you intend to verify; assert with git ls-remote <real remote> refs/heads/<branch>. A rebase onto a newer base also carries upstream .github/workflows/ changes, which a token without workflow scope refuses — move the commits via git bundle to a host whose token has it rather than re-provisioning credentials on production: references/adapting-the-staging-script-to-a-real-service.md. 🔴 A runner label is not a fixed machine size. ubuntu-24.04-arm is 4 vCPU/16 GB on a PUBLIC repo and 2 vCPU/7.7 GB on a PRIVATE one, so moving a workflow between repos can silently halve the build's memory and kill it with exit 143 (SIGKILL/OOM) and no compiler error. Print nproc + free -h in an early step and compare against the measured build peak — details and the sub-package-failure trap in references/ci-offload-build-and-stage.md.

🔴 When your own assert fails a build, suspect the assert before the artifact. Two failed rounds on the same check means stop and switch to a BEHAVIOURAL gate — make the program do the thing (open the DB, serve the request) rather than inspecting what its compiled output looks like. Four CI builds were burned iterating a static bundler check against an artifact that was correct every time, while the substance signals stayed green in every failed run. Fixtures you write yourself cannot falsify your own mental model; only a known-bad and a known-good REAL artifact can. Full case, plus the runtime-proof traps (wrapper vs child pid, prebuilt binary filename) and the rule to demote every COPY of a bad check: references/verification-that-discriminates.md.

🔴 A green backup job is not proof of a recoverable service. Ask "what could I NOT reconstruct from the backup alone?" and answer it by enumerating the app's real inputs against the backup ROOT SET — never by reading the job's exit status. Config living outside the data dir is invisible to a positive root assertion, and when a datastore holds encrypted-at-rest secrets (enc:v1: tokens) the key material is part of the backup set BY DEFINITION; ciphertext without its key restores to nothing. Verify by round-trip restore + hash compare, not by reading the manifest. Also covers a stale lock silently skipping forget --prune forever while the job still exits 0: references/backup-restore-completeness-audit.md.

🔴 After the cutover, publish a before/after table on the SAME metrics, and do not revert the workaround in the same change. A performance hack added for the bug you just fixed (a tmpfs RAM disk, a raised IOPS ceiling) stays until the new build is live AND the fix is measured on the new pid — otherwise a rollback lands on an unprotected host. A workaround also DISTORTS measurement: with the DB on tmpfs, device-level iostat read ~0 KB/s while the real demand was a 231 MB/s burst every 3 minutes, so measure the BURST directly before sizing storage down. And predict, before and after, what the fix will NOT improve — references/native-driver-fallback-memory-signature.md.

Then retire the workaround as its OWN change, decided by measurement. The trap is asking "is the workaround faster?" — it usually is, and that is the wrong question. Measured here: tmpfs 32,434 vs EBS 8,382 SQLite ops/sec (3.9x), against real demand of 0.83 writes/sec at the busiest minute in 24h — 10,099x headroom on the slower option, costing +0.033 ms per insert, or 0.0005% of a 5.6 s request. The interleaved A/B protocol, peak-demand measurement, migration traps (remove symlinks before copying or cp follows them onto themselves; stop the sync timer first; assert the unit no longer references the mount), proving the new path is in use via /proc/<child-pid>/fd, and the discipline of NOT claiming a latency win the measurement does not support, are in references/retiring-a-performance-workaround.md, with a runnable scripts/bench_storage.cjs. Removing a workaround usually invalidates capacity provisioned for it — re-derive that sizing, and check the cooldown (EBS modifications lock for 6 hours) so you size once.

🔴 A workaround's assumptions are ENCODED IN YOUR MONITORS — retire them in the same change. This is the step that gets skipped and it is the dangerous one. Rules written during the incident describe the hack as permanent truth, and once the hack is gone they become blind spots pointed at exactly the failure you most need to see. Found live after removing a tmpfs DB mount: a watchdog charter still said "the DB is on tmpfs and gets fully rewritten, so NEVER trust quick_check on the live file" — true while the hack existed, but now it means a real corruption signal gets dismissed as normal. Alongside it, a "never alarm on external/arrayBuffers" rule that existed only to suppress noise from the very bug just fixed, plus threshold checks on a mount that no longer exists and now silently measure nothing. Grep every monitor, charter, runbook, and alert script for the removed component's name before you call the migration done, and treat any suppression rule as expiring with the bug it was written to silence.

🔴 Before a deploy, back up the state DB with the engine that WRITES it, and prove each copy by opening it. Two plausible methods produced corrupt copies on one occasion: the host python3 stdlib sqlite (3.45.1) made a backup of a DB written by better-sqlite3 3.53.3 that both engines then read as SQLITE_CORRUPT, and VACUUM INTO failed the same way. When the app is on a full-rewrite driver (sql.js), every reader also hits torn reads — the live file can even be observed at 0 bytes mid-write without any data being lost. Backups must retry until quick_check=ok AND config-table row counts match. Full pattern, plus the two independent probes for which SQLite driver is actually loaded in the running process, and the rule that a performance workaround stays until the fix is measured live: references/sqlite-backup-under-a-full-rewrite-driver.md, with a ready-to-run scripts/backup_sqlite_multi.cjs (N-way, retry-until-consistent, verifies by opening each copy). Always land one copy off-host and grab the service .env alongside it — config is what the user cannot rebuild.

A migration to a new deploy mechanism can delete the working one. On one occasion a commit titled ci: move owner-specific builds to the ops repo removed standalone-build.yml from the repo — but the new ops repo only held an unfinished container build, while the router still ran an unpacked standalone bundle under releases/. Net effect: no way to produce a shippable artifact without building on the live host, the exact thing that caused a 30-minute fleet outage. Recover the proven workflow from the commit that built the running release (git show <deployed-sha>:.github/workflows/<file>) rather than reconstructing it, note in the commit message that it can be dropped once the new path is proven, and confirm with the user which mechanism is actually live before adopting the new one.

🔴 This recurs, and the second time it recurred as a RESTORE-THEN-DELETE-AGAIN pair. On one occasion, <sha> restored standalone-build.yml and then <sha> ("ci: keep fork deployment automation in the ops repo") deleted it again — so the branch HEAD had no tarball build while the ops repo still shipped only a container. Skimming the log for "was it restored?" gives a false yes. Only the tree at the branch HEAD is authoritative. Before planning any deploy, run these three and state the answers explicitly:

# 1. what does the live unit actually execute?
systemctl --user cat <svc> | grep -E 'ExecStart|WorkingDirectory'
# 2. does the ref you intend to ship contain a build for THAT artifact shape?
git show origin/<ref>:.github/workflows/<file> >/dev/null 2>&1 && echo PRESENT || echo ABSENT
# 3. can the host even CONSUME the new artifact shape?
getent group docker    # empty group => this user cannot run containers unprivileged
gh workflow list -R <owner>/<repo> --all | grep -i disabled

Check 3 is the one that gets skipped. A container artifact is a different deploy architecture, not a drop-in for a symlink swap: it needs group membership, a rewritten unit file, a new rollback story, and a registry pull path. Never fold that migration into a deploy the user framed as "ship the new code" — surface it as its own decision with options and let them choose. Note also that workflows can exist but be disabled_manually, which gh workflow list --all reveals and a tree listing does not.

Verify that commits the live box has but the new branch "lacks" are real losses. git rev-list <newref>..<deployed-sha> listed two on one occasion. One was a feature reapplied during the rebase under different SHAs — confirm by grepping the feature string in the new tree (git show origin/<ref>:<path> | grep -c FEATURE_FLAG), never by SHA ancestry, since a rebase rewrites them. Only the CI workflow was genuinely gone. Report "nothing is lost" only after that per-commit check, and check whether the corresponding runtime config (an env var the feature reads) is still set on the host.

Extend the bundle asserts to cover what THIS deploy ships, then prove the assert works by running it against the currently-deployed bundle — the new strings must be absent there and the pre-existing one present:

FOUND: <OLD_FIX_SYMBOL> ← old fix, already shipped
not found: no such (module|table): dbstat   ← today's change, correctly absent
not found: <NEW_FLAG>             ← today's change, correctly absent

Use grep -rqsF for any assertion string containing regex metacharacters; plain grep would read (module|table) as a pattern and silently match nothing. Validate the edited workflow before pushing: parse the YAML, extract the run: block, and bash -n it.

🔴 An assert validated only against FIXTURES is not validated. Fixtures you write encode the same mental model as the code you wrote, so they agree with each other and are wrong together. On one occasion three revisions of one release gate each FAILED A CORRECT ARTIFACT while passing 18/18 of their own fixture tests — wrong chunk, then a generic webpack helper present in both bundles, then per-file lookup of module ids that are global to the chunk graph. Each round cost a full ~15 min CI build to discover. Rules: control-test in BOTH directions against REAL artifacts (known-bad must FAIL, known-good must PASS); get the known-good artifact in hand rather than iterating blind across CI round-trips, or make the check PRINT the bytes it could not classify; prefer a BEHAVIOURAL gate over asserting compiler internals — module ids and chunk layout are the compiler's business, so gate on "open the database and round-trip a row", not on bundle archaeology; fail closed on unknown; and when an assert fails ask "is the artifact wrong, or is my assert wrong?" before touching the artifact. Also locate build outputs by search, not a remembered path — an assert on build/Release/*.node reported MISSING one line before the module loaded fine from prebuilds/linux-arm64.node. Full worked case: references/verification-that-discriminates.md.

⚠️ CI runner sizing differs between PUBLIC and PRIVATE repos. The same label ubuntu-24.04-arm yields a free 4 vCPU / 16 GB runner on a public repo and 2 vCPU / 7.7 GB on a private one. Moving a workflow from the public app repo to a private ops repo therefore silently halves its memory, and a build that peaks near 15 GB dies at exit 143 (SIGKILL/OOM) ten minutes into "Creating an optimized production build" — with no OOM message, just the exit code. Echo nproc and free -h in an early step so the runner's real size is in the log, and use a larger-runner label (Blacksmith etc.) already proven in that repo.

🔴 Assert native modules BY PATH, and prove they LOAD. A feature-string assert plus a find -name '*.node' | wc -l count both pass while the one native module the app cannot run without is absent — the broken bundle had more .node files than the working one (87 vs 81). Missing a compiled driver usually does not crash the service; it silently drops to a slower fallback engine that degrades days later as an unrelated-looking error. A transitive dep of a native module is part of that module (better-sqlite3 needs bindings, which needs file-uri-to-path), and its absence reports as Cannot find module 'better-sqlite3' even though the directory is present. After any cutover, read the startup ENGINE/DRIVER-selection log lines, not just the ready line — a fallback ladder announces itself immediately above a cheerful "database ready". Full recipe, asserts, and per-PID verification: references/native-module-completeness-in-bundles.md.

🔴 REFINEMENT (one occasion): "missing native module" is often a BUNDLING defect, not an absent file. Verified end-to-end on the live host: the addon was present at v13.0.1, linux-arm64.node loaded by hand, and createRequire resolved it from all five plausible anchors — yet the app logged Cannot find module 'better-sqlite3' and fell to sql.js. Cause: webpack replaced the injectable loader itself with a stub whose only behavior is to throw, because the call site passed the module name through a variable. Do not stop at "is the file on disk" — that check exonerates the wrong suspect. Read the compiled chunk and print the definition of the module id the call site references. The four-step diagnosis (and why a naive grep -c 'require("x")' returns 726 hits and the OPPOSITE conclusion), the fix (keep module names literal at each call site; hoisting them into a constant re-breaks it), why unit tests pass either way, and the four unrelated-looking symptoms one such defect produced — "memory leak", write storm, backup corruption, latency creep — are in references/webpack-externals-stub-and-driver-fallback.md.

🔴 Before promising ANY upgrade, verify what the running process actually executes. A dependency compiled into the interpreter (SQLite, OpenSSL, zlib) is not versioned by the app release: the app version can be identical before and after, the app's own updater will not fix it, and a package manager may have already dropped a patched interpreter on disk while every live process keeps executing the old, deleted inode. That combination makes a host read as "patched" when it is still vulnerable — and makes "run the updater" a false remedy. The per-PID probes (/proc/<pid>/exe on Linux, lsof txt segment on macOS), the (deleted) tell, and the restart-not-upgrade conclusion are in references/runtime-vs-ondisk-version-verification.md. Read it before scoping any remediation as an upgrade — on one occasion it collapsed a four-host coordinated upgrade into four restarts.

Restarting a busy service over SSH needs a detached script, and success means MainPID CHANGED — not exit 0 and not is-active, which still reports active while a unit drains. Foreground ssh host 'systemctl restart X' times out and leaves the unit half-restarted with MainPID unchanged, which reads as a silent no-op. Method, the lifecycle-guard workaround, and the buffered-output gotcha: references/restarting-busy-services-over-ssh.md.

🔴 A deploy artifact can contain a NESTED copy of its own runtime assets, and the nested one is what gets served. A Next.js output: "standalone" bundle unpacks to both .build/next/ and .build/next/standalone/.build/next/ — and systemd's WorkingDirectory points at the nested path. The assembly step can leave that nested copy STALE while the outer one rebuilds cleanly, so a current release serves weeks-old UI code while the release dir name, mtime, BUILD_SHA, and health-endpoint version all read correct. A feature merged before the stale cut renders fine, "proving" the deploy is healthy. On one occasion this hid a missing dashboard provider for three weeks. Compare BUILD_ID at BOTH levels as a standing post-cutover check — mismatch means the served bundle is stale, and re-extracting the same (already-correct) CI artifact into a fresh release dir fixes it with no rebuild. Generalize: verify the path the process actually reads, not the path that shares its name. references/nested-bundle-staleness-and-control-tests.md

🔴 Run every presence/absence probe against a known-GOOD and a known-BAD control before believing it. In that same session grep 'Provider not found' on a page "proved" a provider was missing — but the route is a client-rendered shell, so a valid provider, a working control, and bogus-provider-xyz all returned the byte-identical 629,808-byte response containing that string. The probe had zero discriminating power and produced a wrong finding reported to the user. If good and bad inputs return the same answer, the probe is broken — fix the probe before touching the subject. Identical response sizes across every input are themselves the tell. For client-rendered pages, grep the JS chunks the page references, not its server HTML; prefer probes whose output differs in KIND (0 chunks vs 1) over "contains a string that is always there."

After cutover, prove the SHIPPED CHANGE is live — not merely that the service booted. Health-green plus a moved symlink is provenance, not proof. When the endpoint your fix repairs sits behind session auth, a 401 is zero evidence either way, and "zero errors since restart" is meaningless if the before-window was also zero. Grep the compiled bundle for the guard and run the same grep against the rollback target to show the strings discriminate: references/proving-a-fix-is-live-post-cutover.md.

🔴 Migrations — not code — are the actual one-way door. A symlink flip is NOT a rollback once the new build has booted against the live DB. Snapshot the data store immediately before cutover and restore code+data together. Full pattern, including how to enumerate pending/destructive migrations before you commit: references/database-migrations-and-rollback.md.

🔴 Before acting on any "the database is corrupt" reading, take a SECOND independent copy and re-check. A ?mode=ro URI connection against a live WAL-mode SQLite database fabricates corruption that is not on disk — in the worked case it reported 13 unreadable tables and a ROLLBACK-terminated dump, while integrity_check on the live file returned ok and every table read fine. Repairing from that dump would have destroyed 417 rows of live config to fix a problem that did not exist. Removing ?mode=ro is necessary but NOT sufficient: a busy WAL database still tears reads intermittently through every client (python, better-sqlite3, and the CLI all flapped on the same file minutes apart), so the durable fix is a retry loop with in-process verification, and the settling test is snapshotting db + -wal + -shm together and checking that static copy. The one-step falsifier, measured flap rates, the correct probe, and the .recover / empty-database-passes-every-check traps: references/sqlite-live-wal-false-corruption.md.

🔴 Never cp a live WAL-mode database for backup/persist. When a service moves its SQLite store to a tmpfs RAM disk (systemd mnt-<name>-ramdb.mount) with a .persist copy on real disk, a sync script that cps the live file while the app writes produces a torn copy that reads back as database disk image is malformed. Use the online backup API and verify the copy before it replaces the previous good one: references/durability-mechanism-verification.md. Also size tmpfs against the engine's WRITE AMPLIFICATION, not the file size — outgrowing the mount makes SQLite return exactly a disk I/O error.

⚠️ CORRECTION (one occasion) to the earlier reading of this same incident. A prior pass concluded "the cp corrupted the persist twin, the RAM copy stayed clean." Both files were fine. Snapshots of each passed integrity_check 3/3 — the malformed readings were torn reads of live files, in BOTH directions (persist looked corrupt first, then RAM looked corrupt 20 minutes later). Acting on either reading would have rebuilt a healthy database from another healthy database. The real cause of the user-visible disk I/O error was a missing native SQLite driver in the deployed bundle forcing a sql.js/WASM fallback that rewrites the entire file on every persist — references/native-module-completeness-in-bundles.md. Two standing lessons: when a corruption reading flips direction between runs, the reading is the defect; and a resource-exhaustion symptom can be a downstream effect of a packaging regression, so check which engine the process actually loaded before sizing storage.

🔴 A failing smoke check is a claim about your TEST before it is a claim about the build. Reproduce the check's mechanism in isolation before reporting a defect. This session declared a good artifact broken because the streaming probe piped curl into head -c 300: head closes the pipe, curl dies of SIGPIPE, and the server logs disconnect: request_signal_aborted — which reads exactly like a server bug. The same request captured to a file returned 1602 bytes and 8 SSE events. Never pipe a stream into head; read -n and an early break in a while-read loop kill the producer the same way. Ask "would this check pass against the known-good build currently running?" before escalating. Two sibling artifacts bit the same session — EXIT=28 is curl timing out under staging load (not the service dying), and a result variable computed but never read makes a gate that cannot fail. All three, with the falsifiers: references/smoke-test-and-staging-script-artifacts.md.

Verify the fix by re-running, not by explaining. Twice this session a plausible cause was announced as resolved before retest: the ?mode=ro removal (the next run failed identically) and a driver reorder justified by ten trials of an intermittent fault. Both were genuine improvements; neither was the cause. Ten trials cannot characterize a flapping failure — do not reorder logic on that evidence and call it proven. When a fix lands, re-run the failing operation end to end and quote the new output; when the retest contradicts you, retract in the first line rather than stacking a second theory on the first.

Scheduled DB retention pruning follows the same stop→mutate→restart discipline. the router's ~64 MB in-memory SQLite page cache means external deletes while the router is running silently reappear on flush — the mutation sequence must be stop→prove gone→delete+VACUUM→start, with a verified online backup taken while the service is still healthy. Full playbook including the 12-table retention schema with per-table timestamp column/format mappings, the better-sqlite3-only constraint, the recovery/rollback procedure, and the cron-job creation pattern (one-shot validation → weekly reschedule): references/sqlite-retention-maintenance.md.

Before you conclude a red CI check is yours, check out the pristine base ref and run the same gate with your change absent. Bookkeeping gates (frozen file sizes, changelog presence, lint-warning counts) fail for whoever pushes next, not whoever caused the drift — and an active release branch is often already red from the maintainer's own same-day commits: references/ci-failure-attribution-base-vs-branch.md.

🔴 Branch from the commit you actually RUN, not the checkout's HEAD or the latest tag. These routinely disagree: a deploy host's checkout said 3.8.49 while the symlinked artifact reported 3.8.50, built from a sha that did not even exist in that checkout. Resolve it from the running service outward — health-endpoint version → current symlink → BUILD_SHA → git branch -a --contains <sha> — and verify the true upstream with gh repo view --json parent rather than typing a remembered URL. Also: before adding a "missing" setting, trace all six links of its chain (type → default → persistence → startup application → runtime re-application → UI/API); a setting that appears absent is often present and broken, and shipping a second mechanism beside a defective one makes it worse. references/config-setting-chain-and-fork-prs.md

🔴 Fork patch custody is a release invariant, not branch trivia. Before cleaning or rebasing a fork, inventory fork-only behavior across all branches and prove every production-critical semantic patch is carried by the active release. Then verify the built artifact and running process separately: an environment variable can remain configured while its reader is absent from the deployed bundle. Cherry-picks also break original-SHA ancestry, so use symbols, focused tests, patch identity, and carrier commits rather than merge-base --is-ancestor alone. Full procedure: references/fork-patch-custody-and-live-proof.md.

Worked example: references/router-outage-postmortem.md — a ~30-minute fleet-router outage that hit every failure mode below at once, plus how Hermes clients retry/fall back when their router dies (SDK max_retries: 0 is deliberate; policy lives in the outer conversation loop) and how to tell a merely idle downstream client from a genuinely broken one.

Read the host's own runbook before you build

And load this skill before you plan the deploy, not after the plan is written. On one occasion a full cutover plan was drafted from first principles — backup, unpack, flip, verify — and only when the user asked "there is a skill for this, are you using it?" did the session load it. The skill already carried a proven scripts/stage_and_smoke.sh with a test-port staging step against a DB copy that the hand-written plan omitted entirely; that step then caught a smoke-gate defect before anything was promoted. If the task matches a skill's trigger, load it before designing the approach: skills_list is cheap and the packaged scripts encode failures you will otherwise rediscover live.

Before diagnosing or building anything on a host you did not configure, look for operator docs on the box: /root/CLAUDE.md, ~/CLAUDE.md, ~/README, ~/scripts/, ~/*-runbook.md, and any watchdog's journal.

In the one occasion outage the host carried a BUILD GOTCHAS section naming the exact trap that caused the incident (Turbopack OOMs the box; use webpack), plus a ready-made ~/build-clean.sh. It was never read. Three documented rules were violated and the fleet went down.

Corollaries:

• A project's own release script may ignore its own env guard. .env set APP_USE_TURBOPACK=0, yet npm run build:release still invoked next build --turbopack. Verify the tool actually used by grepping the build log — assert it in CI so it can never regress.
• Check for a supervising watchdog before doing anything unusual. This host ran an agent every 15 min that killed rogue builds and paged the owner. It correctly caught and killed the first build. A setsid nohup detached second build then escaped the pkill and caused the outage — the "clever" detachment defeated the safety system. Detach to survive an SSH pipe drop, but never to escape supervision; prefer the host's own build runner.
• Check the cloud auto-recovery path before claiming credit for a fix. A CloudWatch status-check alarm had already rebooted the box; the manual aws ec2 reboot-instances was not what recovered it.

Diagnose from evidence on the box, not from inference

In that incident three successive root causes were asserted and published into a skill before any measurement: "t4g CPU-credit exhaustion" (wrong — the host is non-burstable), "vCPU saturation starved the process" (wrong — CPU was 32% idle at the worst moment), and only then the truth: memory exhaustion → page-cache eviction → 100% swap → ~1940 pages/s thrash.

The data was sitting on the box the whole time. sysstat keeps 10-minute samples, so post-mortems are almost always possible:

sar -r -f /var/log/sysstat/sa<DD> -s 22:00:00 -e 23:00:00   # memory
sar -S -f /var/log/sysstat/sa<DD>... # swap used
sar -W -f /var/log/sysstat/sa<DD>... # swap in/out rate
sar -u -f /var/log/sysstat/sa<DD>... # cpu (%nice = build)
sar -q -f /var/log/sysstat/sa<DD>... # load + blocked
journalctl -k -b -1 | grep -i "out of memory\|killed process"

Distinguishing detail: no OOM kill occurred. The kernel never killed anything, it just made everything unusably slow — worse than an OOM kill, which would have killed the build and let the service recover unattended. "No OOM in dmesg" does NOT rule out a memory failure.

Rule: state a root cause only after a measurement supports it. Say "I don't know yet, pulling sar" instead of naming a plausible mechanism. A wrong diagnosis written into a skill outlives the incident and misleads the next session.

Before you touch anything: the four preflight facts

Gather these first. Each one has burned a real deploy.

1. How is the service actually supervised?

   systemctl --user is-active <svc>     # USER units are easy to miss
   systemctl is-active <svc>            # system level
   systemctl --user cat <svc>           # read WorkingDirectory + ExecStart
   ps -o pid,ppid,cmd -p <pid>          # ppid → /usr/lib/systemd/systemd --user ?
   launchctl list | grep -i <svc>       # macOS
   

   A system-level is-active returning inactive does NOT mean unsupervised. The unit may live in the user manager. Getting this wrong leads to a false "nothing will restart this if it dies" alarm — and to missing the fact that the unit's WorkingDirectory is about to be deleted by your build.

2. What does the build command actually do first? Read the package script.

   grep -A2 '"build:release"' package.json
   

   If it starts with rm -rf <dir> and <dir> is (or contains) the supervisor's WorkingDirectory, an in-place build deletes the running service's filesystem out from under it. The process survives on memory-mapped code, so everything looks fine — until the next restart, which cannot succeed. Your rollback window closed silently the moment the build started.

3. What is the host's real capacity, and is it shared with the service?

   nproc; free -m; uptime
   aws ec2 describe-instances --filters "Name=ip-address,Values=<ip>" \
     --query 'Reservations[*].Instances[*].[InstanceId,InstanceType,State.Name]' --output table
   

   Do not reach for the "burstable instance exhausted its CPU credits" explanation without confirming the instance is burstable. On a non-burstable box (Graviton r8g/m8g, c6a, etc.) there is no credit mechanic at all — plain 2-core saturation is enough to starve both the service and sshd. Right symptom, wrong cause, wrong fix.

4. Do you have a rollback you have actually tested? See below.

Snapshot before, always

TS=$(date +%Y%m%dT%H%M%SZ)
git rev-parse HEAD > ~/rollback-HEAD-$TS.txt
cp.env ~/rollback-env-$TS.bak
systemctl --user cat <svc> > ~/rollback-unit-$TS.service
tar czf ~/rollback-build-$TS.tar.gz.build dist # the artifact itself
# plus a DB/data backup if the upgrade runs migrations

Migrations matter: if the new version applies schema migrations, a code rollback alone is not sufficient — the data may no longer be readable by the old binary. Snapshot the DB whenever the release notes mention migrations.

The deploy pattern: build into a NEW directory, flip a symlink

This is the structural fix. It makes "the service has no working directory" impossible by construction.

srv/releases/v1.2.3-<sha>/    <- build output lands here
srv/current -> releases/v1.2.3-<sha>/

Point the supervisor's WorkingDirectory at current, never at a real build dir.

REL=~/srv/releases/v1.2.3-$(git rev-parse --short HEAD)
mkdir -p "$REL"
# ... build into $REL, or unpack a prebuilt artifact into it ...
ln -sfn "$REL" ~/srv/current          # atomic flip
systemctl --user restart <svc>        # seconds of downtime

Rollback = flip the symlink to the previous release dir and restart. Keep the last 2–3 releases. There is never a moment where the live service has no directory to run from.

Build on a different host when the build is CPU-heavy

The symlink solves "no workdir." It does not solve CPU starvation — a Next.js/webpack/Turbopack compile will still saturate a 2-core box and stall the service sitting next to it. For heavy builds, build elsewhere and ship the artifact:

# on a BUILD host — MUST match the target's OS and arch
npm ci && npm run build:release
tar czf app-build.tar.gz.build dist

# on the live host: unpack into a NEW release dir, then flip
scp app-build.tar.gz user@prod:/tmp/
ssh user@prod 'set -e; cd ~/srv
  REL=releases/v1.2.3-$(date +%Y%m%dT%H%M%SZ); mkdir -p "$REL"
  tar xzf /tmp/app-build.tar.gz -C "$REL"
  ln -sfn "$PWD/$REL" current
  systemctl --user restart <svc>'

Choosing a build host: native modules make this OS+arch-specific

You cannot build a Linux artifact on macOS just because both are arm64. Native addons compile or vendor per-platform:

find.build -name '*.node' | wc -l # count native binaries
ls node_modules/@img                        # sharp ships per-platform variants
find. -path '*better-sqlite3*' -name '*.node'

better-sqlite3 compiles against the build platform; sharp installs only the matching sharp-<os>-<arch> package. A macOS build produces sharp-darwin-arm64 and a Darwin better_sqlite3.node — both fail to import on a Linux target. Match uname -sm between build host and target. Match the Node major version too (ABI compatibility for native addons).

Detach long builds from SSH

A build run as ssh host 'npm run build' dies when the SSH pipe drops — and the pipe will drop precisely when the box gets starved, i.e. exactly when the build is mid-write. That leaves a half-deleted, half-written artifact.

ssh host 'cd ~/app && setsid nohup npm run build:release \
  > /tmp/build.log 2>&1 < /dev/null &'

Then poll from separate short-lived SSH calls. Never hold a long-lived SSH session open as the build's parent process. Prefer background=true + notify_on_complete=true for the poller rather than blocking.

When you lose the box mid-deploy

Scope it from outside first. Distinguish "process starved" from "host dead" before reaching for a reboot:

import socket
for port in (443, 22):
    s = socket.socket(); s.settimeout(10)
    try: s.connect(("<ip>", port)); print(port, "OPEN")
    except Exception as e: print(port, e)
    finally: s.close()

Both ports open but HTTP hanging ⇒ the reverse proxy is accepting and the app process is starved. Ports refused/timing out at TCP ⇒ host or network problem.

Keep an out-of-band health poller running so you learn the instant it recovers, rather than guessing:

for i in $(seq 1 60); do
  code=$(curl -s -m 15 -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $KEY" https://host/healthz)
  echo "[$(date +%H:%M:%S)] http=$code"
  [ "$code" = "200" ] && { echo RECOVERED; break; }
  sleep 15
done

Cloud reboot when the box is unreachable and the build is unkillable:

aws ec2 reboot-instances --instance-ids <id> --region <region>

Note reboot-instances is plural; reboot-instance is not a valid subcommand and prints the entire subcommand list at you. Find the instance from its IP with describe-instances --filters "Name=ip-address,Values=<ip>".

Expect a recovery ladder, don't panic partway: 000 (nothing listening) → 502 (reverse proxy up, app still booting) → 200. Allow ~2 minutes before declaring the restore failed.

Verification: exit 0 is not proof

After restoring or deploying, verify in this order — each step catches something the previous one cannot:

1. Supervisor state, including restart count:

   systemctl --user show <svc> -p ActiveState,SubState,NRestarts,ExecMainStartTimestamp
   

   NRestarts=0 means a clean start. A nonzero count means it crash-looped its way back up — technically "active," actually broken.
2. Port listening — ss -tlnp | grep <port>.
3. Health endpoint, probed 2–3× (handlers often lazy-load on cold start).
4. A real end-to-end request that does actual work, not just a liveness ping. A router must serve a real inference; an API must return real data.
5. The UI/dashboard surface, if the app has one. An API-only smoke test passes happily while the web UI is broken — in a Next.js/SSR app the pages are a separate rendering path from the API routes, so bad static-asset paths or a failed server render show up nowhere in an API check. Assert the page returns real markup, not an error page, and that a static chunk actu

…(truncated)