Strays

Strays: processes that outlived what started them

Read this first

When several unrelated things start failing at once, the cause is almost never several unrelated bugs. It is one shared resource.

CPU, memory, disk, file descriptors, ports, a lock. Check the shared resource before debugging any individual failure.

This matters more for an agent than for a person, because the instinctive response to a timeout — retry it — spawns another process and makes the real problem worse. Two retries of a hung test suite is two more permanent CPU burners.

The symptom

Failures that arrive together and make no sense individually:

• commands that worked five minutes ago now time out
• tests fail with no code change, differently each run
• builds hang partway with no error
• port already in use when nothing should be listening
• background tasks report failure with empty output

The tell is breadth. One flaky test is a flaky test. Six unrelated things failing in ten minutes is a machine problem.

Step 1: look at load before looking at code

scripts/strays.py

The first line is the one that matters:

  load 69.90 / 4 cores -- OVERLOADED -- almost everything will fail or hang

Load is the number of processes wanting CPU. Compare it against core count, not against zero:

If the ratio is above ~1.5, stop debugging the failure. It is downstream.

Step 2: find what is holding the machine

strays.py flags a process when it is either a tool that should have finished, or an unrecognised process that is both orphaned and burning CPU:

  6 stray process(es), 316% CPU between them:

    pid 111503    51.8%   2h35m  python -m pytest -q -x
                 why: 52% CPU for 2h35m; orphaned (parent exited)

Being orphaned is not by itself suspicious — daemons are reparented to init by design. A pytest that has been running for two and a half hours is.

It also lists dev-server ports still held, which is where address already in use comes from.

Step 3: clean up, having looked

scripts/strays.py --kill

It reports first and kills nothing without that flag, because the judgement is not automatable: a long compile is not a stray. Read the list. SIGTERM goes first, then SIGKILL to whatever ignored it, and always to the whole process group.

It never touches itself, its own ancestors, or systemd, sshd, docker, tmux, claude — killing a parent of the current session would end the session.

Load falls slowly; it is an average. Judge recovery by the process list, not the number.

Step 4: fix the source, or it comes back

Cleaning up is not the fix. Find which spawn leaked and repair it, or the same processes return on the next run.

The overwhelmingly common cause is killing a shell instead of a process tree:

# Leaks. shell=True runs /bin/sh -c "pytest ..."; the timeout kills the shell
# and orphans pytest, which keeps running -- forever, if the code it is testing
# happens to loop.
subprocess.run(cmd, shell=True, timeout=30)

# Correct: own process group, and kill the group on timeout.
proc = subprocess.Popen(cmd, shell=True, start_new_session=True)
try:
    proc.wait(timeout=30)
except subprocess.TimeoutExpired:
    os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
    proc.wait(timeout=10)

references/spawning.md has the equivalent for Node, Go, Rust and bash, plus cleanup on interrupt and the double-fork case.

Rules for spawning anything

1. Every spawn gets a timeout. No exceptions — a hang with no timeout is a leak waiting for the parent to die.
2. Every timeout kills the group, not the process. start_new_session=True plus killpg. Killing the direct child is the bug, not the fix.
3. Reap what was killed. wait() afterwards, or it lingers as a zombie.
4. Clean up on the way out too — try/finally, or a signal handler. Ctrl-C is the most common way a run ends, and the least tested.
5. Prefer a list argv over shell=True. No shell means no intermediate process to lose track of.
6. Long-running servers get a recorded PID and an explicit stop, not a hope that something will tidy up later.

Before ending a long session

Run scripts/strays.py once. A session that started dev servers, watchers or test runs has probably left something behind, and the next session inherits a slower machine with no idea why.

Resources

• scripts/strays.py — reports load against core count, finds abandoned processes with reasons, lists held ports. --kill to clean up, --json for scripting. Refuses to touch itself, its ancestors, or system processes.
• references/spawning.md — leak-proof spawning in Python, Node, Go, Rust and bash; process groups explained; interrupt handling; container and CI notes.