rust-build-hygiene — launch builds that die with you and clean up after themselves
Orphaned Rust builds outlive their launcher, hold target-dir locks (stalling other builds 30+ min), burn cores, and leave tens of GB of stale target/ behind. Every instance traces to the same root cause: a build launched in a way that survives the process that started it. (cpu-reduction is the mop; this is "don't spill.")
The one rule: NEVER detach a build
A detached build reparents to PID 1 the moment its launcher exits, so it outlives the agent/session/turn, keeps holding the cargo target-dir lock, and TaskStop does NOT reap it (TaskStop kills the agent, not its background bash jobs). This is the most common orphan and the usual cause of Blocking waiting for file lock on artifact directory on the next build.
# BANNED — these orphan to PID 1 and survive TaskStop:
setsid cargo build ... &
nohup cargo build ... &
cargo build ... & disown
How to run a build correctly, by situation
| Situation |
Do this |
Why |
| Quick interactive build/test (< a few min) |
Foreground Bash call (through scripts/rust-build.sh) |
Dies with the turn; the harness caps foreground at ~10 min |
| A build you want to keep working alongside |
Bash with run_in_background: true |
Harness-TRACKED — reaped when the session ends, shows in the background-jobs list; NOT detached |
| A long build you will REST on until it finishes |
Dispatch a sub-agent that runs the build in ITS OWN foreground and returns the result |
The sub-agent's liveness is what the harness tracks. Never rest on a bare background shell |
| A long build in CI / on a VM |
scripts/ci-watch.ts (the ci-watch skill) or a sub-agent owning the watch |
Own the wait in a tracked process, never a detached poll loop |
Never fake-wait on a build with a detached shell + a sleep/poll loop.
The fleet, not your build, is what saturates the host
A per-build cap cannot bound N builds. Every build can be individually blameless — --profile fast, QoS-clamped, jobs = 6 — and the machine still dies, because the caps multiply instead of adding. Measured 2026-08-19: 13 concurrent agent builds, every one of them already on --profile fast, produced a 78-way oversubscription of 10 cores — load 464, 0% idle, 36% sys, and a reqwest compile that normally takes ~30s taking 28 minutes. Nothing was misconfigured. There was simply no cap on the SUM.
make qos-global installs the global governor and is what actually bounds the fleet. It registers scripts/rustc-qos.sh as the machine-wide rustc wrapper, where it does three jobs: clamp QoS; let at most TWO builds compile at a time (NUB_BUILD_SLOTS, default 2 — every other build's first rustc waits in a first-come-first-served queue until a holder's cargo exits, dies, or goes idle for NUB_BUILD_IDLE=120s, so a cargo test running its tests or a cargo blocked on a target lock does not hold the machine); and across the compiling builds, hold one of NUB_RUSTC_LIMIT (default 6) tokens for the life of each rustc. Two builds over six tokens bounds the memory peak to 6 big-crate compiles (12 GiB) and keeps a second build's worth of cores busy; strict one-at-a-time was the first cut (2026-08-28) and was measured idling nine cores behind one starved compile while eight builds queued 25 minutes. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script. A build that queues is not stuck — after 20s it prints one rustc-qos: this build is queued … line on its cargo's stderr, and make build-status shows the holders and the queue. rust-analyzer is exempt so the editor never waits behind agent builds.- Never blank
RUSTC_WRAPPER, and never set NUB_BUILD_FG=1 from an agent. Blanking is cargo's documented "no wrapper", and it opts the build out of the global cap; scripts/rust-build.sh used to do exactly this, which is why 10 of those 13 builds were ungoverned. NUB_BUILD_FG=1 opts a build out of the QoS clamp and the build-slot queue (a bare cargo still takes rustc tokens; through rust-build.sh it blanks both wrapper keys, so out of the tokens too) — it exists for a HUMAN at a terminal whose build must not wait behind the fleet, and an agent that sets it recreates the 2026-08-19 incident. NUB_BUILD_SLOTS=0 disables only the slot layer and NUB_BUILD_SLOTS=1 restores strict one-at-a-time — both are PER-PROCESS environment knobs, read by the wrapper of the cargo that inherits them, not host-wide settings. The host-wide switch is make build-slots-off / build-slots-on (a file every wrapper checks each second, so it also releases builds already queued); it is for an emergency, and it leaves the QoS clamp and the tokens in place.
make build-status answers "why is this machine saturated?" and "why is my build not starting?" It prints the sum no single session can see: load, which builds hold the compile slots and who is queued behind them (each tagged with its worktree), token occupancy, a STALE WRAPPER line when an older checkout's make install-dev downgraded the governor, and which builds are outside the cap. Run it before concluding your own build is slow or hung — a build whose rustc sits at Compiling for minutes with no CPU is queued, not broken. A foreground Bash call whose cargo goes silent at Compiling is the same thing: relaunching it puts the new cargo at the BACK of the queue.
Performance — reuse the cache, clamp the QoS, cap the jobs
- Build through
scripts/rust-build.sh (drop-in for cargo). It picks the right target dir (shared by default, auto-isolates when a worktree diverges a depended-on crate) and applies a darwin QoS clamp (taskpolicy -c utility) plus a job cap on big hosts (CARGO_BUILD_JOBS = ncpu-4). That job cap bounds ONE build; make qos-global is what bounds the fleet.
- Use the
fast profile to iterate (--profile fast → target/fast/nub, ~5s incremental), never release (its lto=thin + codegen-units=1 re-LTOs the whole binary every change).
- One target dir per CONCURRENTLY-building tree. Two builds on one target dir serialize on cargo's lock — that IS the contention. A serial multi-phase epic reuses ONE dedicated warm target across its phases; never point two concurrent builds at it. See
rust-build.
- sccache does nothing here (measured 0% cross-worktree hit — it keys on the rustc command line, which embeds the absolute target path). A stable per-tree target dir is the whole answer.
Self-cleaning
- A worktree owns its target.
git worktree remove <path> --force drops the worktree; rm -rf <path>-target drops its private target dir. Do both when the work lands. The shared ~/.cache/nub/shared-target is intentionally left for the next worktree.
- A sub-agent that built in an isolated target cleans it up on completion — unless a serial chain will reuse it (then hand the warm target forward explicitly). Say which in the dispatch prompt.
- Prune stale worktrees periodically.
git worktree list → remove dead ones → git worktree prune. The worktree skill owns the lifecycle; cpu-reduction §2b has the disk-pressure sweep.
- Never
cp -r the repo to isolate a build — the tree carries multi-GB target//.repos// node_modules. Use git worktree add or git clone --depth 1 file://$PWD + a private CARGO_TARGET_DIR.
If it already orphaned
cpu-reduction §2: ps ... | grep -Ei 'rustc|cargo|lld', find the detached build holding the lock, pkill -f '<target-dir>' (artifacts persist = still warm), hand the contention-free target to ONE fresh foreground build.
1---2name: rust-build-hygiene3description: Best practices for spinning up nub Rust builds so they are PERFORMANT and CLEAN THEMSELVES UP — the prevention side of the recurring orphaned-build problem on the maintainer's dev host. Invoke (via the Skill tool) before launching any `cargo build`/`test`/`clippy` you might background or leave running, when setting up a build in a sub-agent, or when deciding how to wait on a long build. Encodes the ONE rule that stops the bleeding — never DETACH a build (setsid/nohup/`& disown` reparent it to PID 1, it outlives its launcher, holds the target-dir lock for 30+ min, and `TaskStop` does NOT reap it) — plus how to background correctly (harness-tracked), how to wait on a long build (a sub-agent that owns the wait, never a detached shell), one-target-per- concurrent-build, the fast profile + QoS clamp, and cleanup-on-done. For clearing residue that already accumulated, see `cpu-reduction`; for the target- dir sharing/isolation decision, see `rust-build`; for the worktree loop, `dev-loop`.4---56# rust-build-hygiene — launch builds that die with you and clean up after themselves78Orphaned Rust builds outlive their launcher, hold target-dir locks (stalling other builds 30+ min), burn cores, and leave tens of GB of stale `target/` behind. Every instance traces to the same root cause: a build launched in a way that survives the process that started it. (`cpu-reduction` is the mop; this is "don't spill.")910## The one rule: NEVER detach a build1112A detached build **reparents to PID 1** the moment its launcher exits, so it outlives the agent/session/turn, keeps holding the cargo target-dir lock, and **`TaskStop` does NOT reap it** (TaskStop kills the agent, not its background bash jobs). This is the most common orphan and the usual cause of `Blocking waiting for file lock on artifact directory` on the next build.1314```sh15# BANNED — these orphan to PID 1 and survive TaskStop:16setsid cargo build ... &17nohup cargo build ... &18cargo build ... & disown19```2021## How to run a build correctly, by situation2223| Situation | Do this | Why |24| --- | --- | --- |25| Quick interactive build/test (< a few min) | Foreground `Bash` call (through `scripts/rust-build.sh`) | Dies with the turn; the harness caps foreground at ~10 min |26| A build you want to keep working alongside | `Bash` with `run_in_background: true` | Harness-TRACKED — reaped when the session ends, shows in the background-jobs list; NOT detached |27| A long build you will REST on until it finishes | Dispatch a **sub-agent** that runs the build in ITS OWN foreground and returns the result | The sub-agent's liveness is what the harness tracks. Never rest on a bare background shell |28| A long build in CI / on a VM | `scripts/ci-watch.ts` (the `ci-watch` skill) or a sub-agent owning the watch | Own the wait in a tracked process, never a detached poll loop |2930Never fake-wait on a build with a detached shell + a `sleep`/poll loop.3132## The fleet, not your build, is what saturates the host3334**A per-build cap cannot bound N builds.** Every build can be individually blameless — `--profile fast`, QoS-clamped, `jobs = 6` — and the machine still dies, because the caps multiply instead of adding. Measured 2026-08-19: 13 concurrent agent builds, every one of them already on `--profile fast`, produced a 78-way oversubscription of 10 cores — load 464, 0% idle, 36% sys, and a `reqwest` compile that normally takes ~30s taking 28 minutes. Nothing was misconfigured. There was simply no cap on the SUM.3536- **`make qos-global` installs the global governor and is what actually bounds the fleet.** It registers `scripts/rustc-qos.sh` as the machine-wide rustc wrapper, where it does three jobs: clamp QoS; **let at most TWO builds compile at a time** (`NUB_BUILD_SLOTS`, default 2 — every other build's first rustc waits in a first-come-first-served queue until a holder's cargo exits, dies, or goes idle for `NUB_BUILD_IDLE`=120s, so a `cargo test` running its tests or a cargo blocked on a target lock does not hold the machine); and across the compiling builds, hold one of `NUB_RUSTC_LIMIT` (default 6) tokens for the life of each rustc. Two builds over six tokens bounds the memory peak to ~6 big-crate compiles (~12 GiB) and keeps a second build's worth of cores busy; strict one-at-a-time was the first cut (2026-08-28) and was measured idling nine cores behind one starved compile while eight builds queued 25 minutes. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script. A build that queues is not stuck — after 20s it prints one `rustc-qos: this build is queued …` line on its cargo's stderr, and `make build-status` shows the holders and the queue. `rust-analyzer` is exempt so the editor never waits behind agent builds.37- **Never blank `RUSTC_WRAPPER`, and never set `NUB_BUILD_FG=1` from an agent.** Blanking is cargo's documented "no wrapper", and it opts the build out of the global cap; `scripts/rust-build.sh` used to do exactly this, which is why 10 of those 13 builds were ungoverned. `NUB_BUILD_FG=1` opts a build out of the QoS clamp and the build-slot queue (a bare `cargo` still takes rustc tokens; through `rust-build.sh` it blanks both wrapper keys, so out of the tokens too) — it exists for a HUMAN at a terminal whose build must not wait behind the fleet, and an agent that sets it recreates the 2026-08-19 incident. `NUB_BUILD_SLOTS=0` disables only the slot layer and `NUB_BUILD_SLOTS=1` restores strict one-at-a-time — both are PER-PROCESS environment knobs, read by the wrapper of the cargo that inherits them, not host-wide settings. The host-wide switch is `make build-slots-off` / `build-slots-on` (a file every wrapper checks each second, so it also releases builds already queued); it is for an emergency, and it leaves the QoS clamp and the tokens in place.38- **`make build-status` answers "why is this machine saturated?" and "why is my build not starting?"** It prints the sum no single session can see: load, which builds hold the compile slots and who is queued behind them (each tagged with its worktree), token occupancy, a STALE WRAPPER line when an older checkout's `make install-dev` downgraded the governor, and which builds are outside the cap. Run it before concluding your own build is slow or hung — a build whose rustc sits at `Compiling` for minutes with no CPU is queued, not broken. A foreground Bash call whose cargo goes silent at `Compiling` is the same thing: relaunching it puts the new cargo at the BACK of the queue.3940## Performance — reuse the cache, clamp the QoS, cap the jobs4142- **Build through `scripts/rust-build.sh`** (drop-in for `cargo`). It picks the right target dir (shared by default, auto-isolates when a worktree diverges a depended-on crate) and applies a darwin QoS clamp (`taskpolicy -c utility`) plus a job cap on big hosts (`CARGO_BUILD_JOBS = ncpu-4`). That job cap bounds ONE build; `make qos-global` is what bounds the fleet.43- **Use the `fast` profile to iterate** (`--profile fast` → `target/fast/nub`, ~5s incremental), never `release` (its `lto=thin` + `codegen-units=1` re-LTOs the whole binary every change).44- **One target dir per CONCURRENTLY-building tree.** Two builds on one target dir serialize on cargo's lock — that IS the contention. A serial multi-phase epic reuses ONE dedicated warm target across its phases; never point two concurrent builds at it. See `rust-build`.45- **sccache does nothing here** (measured 0% cross-worktree hit — it keys on the rustc command line, which embeds the absolute target path). A stable per-tree target dir is the whole answer.4647## Self-cleaning4849- **A worktree owns its target.** `git worktree remove <path> --force` drops the worktree; `rm -rf <path>-target` drops its private target dir. Do both when the work lands. The shared `~/.cache/nub/shared-target` is intentionally left for the next worktree.50- **A sub-agent that built in an isolated target cleans it up on completion** — unless a serial chain will reuse it (then hand the warm target forward explicitly). Say which in the dispatch prompt.51- **Prune stale worktrees periodically.** `git worktree list` → remove dead ones → `git worktree prune`. The `worktree` skill owns the lifecycle; `cpu-reduction` §2b has the disk-pressure sweep.52- **Never `cp -r` the repo to isolate a build** — the tree carries multi-GB `target/`/`.repos/`/ `node_modules`. Use `git worktree add` or `git clone --depth 1 file://$PWD` + a private `CARGO_TARGET_DIR`.5354## If it already orphaned5556`cpu-reduction` §2: `ps ... | grep -Ei 'rustc|cargo|lld'`, find the detached build holding the lock, `pkill -f '<target-dir>'` (artifacts persist = still warm), hand the contention-free target to ONE fresh foreground build.
nubjs