Shell Process Patterns

Shell Process Patterns

Start, supervise, and terminate shell processes safely -- background jobs, subshells, signal handlers, and cleanup. The dominant failure mode in this domain is silent state: a process that looks killed but still holds a port, a trap that looked fine but never fired on the child, a set -e script that kept running because || true swallowed the error. This skill picks the right pattern, implements it with the real PID (not the wrapper), and verifies the observable state afterward.

Scope

In scope:

• Starting background processes (&, nohup, disown, setsid, daemonization).
• Capturing the actual child PID and reconciling it against observed system state.
• Signal handling -- SIGTERM vs SIGKILL, exec semantics, process groups, subshell inheritance.
• Trap discipline -- EXIT vs signal traps, ordering, inheritance in subshells and functions.
• Cleanup verification -- kill-and-check, not kill-and-assume.
• set -e / set -u / set -o pipefail interactions and the || escape hatch.
• wait semantics, reaping children, race conditions between background-start and resource readiness.

Out of scope:

• Cron scripts and scheduled job reliability (owned by cron-job-auditor).
• Polling, retry, backoff, health-check loops (owned by condition-based-waiting).
• Service health reporting (owned by service-health-check).
• Fish shell configuration (owned by shell-config).
• Shell language features unrelated to process lifecycle -- parameter expansion, arrays, etc.

Reference Loading Table

Instructions

Before implementing any pattern, read the repository CLAUDE.md and search the codebase for existing process-management patterns so the new code matches what is already there. Consistency with existing scripts beats local optimization.

Step 1: Pick the pattern

Walk this decision tree. Pick exactly one pattern per task -- do not pre-emptively wrap a background process in a daemon, and do not add a trap handler for a script that runs for 50ms.

1. Are you starting a new process?
   YES -> Is the parent going to exit before the child?
          YES -> Daemonization (Step 3, load references/starting-processes.md)
          NO  -> Background start (Step 2, load references/starting-processes.md)
   NO  -> Continue

2. Do you need to kill or inspect a process someone else started?
   YES -> PID resolution (Step 4, load references/pid-resolution.md)
   NO  -> Continue

3. Are you writing a supervisor (script that manages children)?
   YES -> Signal + trap discipline (Step 5, load references/signals-and-traps.md)
   NO  -> Continue

4. Are you finishing a destructive operation (kill, rm, release)?
   YES -> Cleanup verification (Step 6, load references/cleanup-verification.md)
   NO  -> Stop. The task may not belong in this skill.

Step 2: Start a background process

A background process in the same session (terminal open, parent stays alive). Load references/starting-processes.md for full rationale.

Minimum discipline:

1. Redirect all three fds. cmd > log 2>&1 < /dev/null & -- because an un-redirected background process inherits the terminal, and stray stdin reads block forever.
2. Capture the PID defensively. $! is the last backgrounded job's shell-level PID. If you wrap in nohup, you get the nohup PID, not the child. Re-query before acting on it (Step 4).
3. Decide if disown is needed. disown $! removes the job from the shell's job table so the shell does not send SIGHUP when it exits. Needed for scripts that start long-running children and return.

Minimal correct pattern (in-session, parent stays alive):

cmd > /tmp/cmd.log 2>&1 < /dev/null &
pid=$!
kill -0 "$pid" 2>/dev/null || { echo "failed to start" >&2; exit 1; }

Constraint: never use cmd & with no redirection in a non-interactive script -- because inherited stdout can deadlock when the terminal closes, and inherited stderr pollutes the parent's log.

Gate: run kill -0 $pid (no-op signal, checks existence) before treating the PID as valid -- because $! may name a process that died in the first millisecond (misspelled command, missing binary) and the script will happily kill a nonexistent PID later.

Step 3: Daemonize a process

Daemonization is needed when the child must outlive the parent session (SSH logout, terminal close, script exit). Load references/starting-processes.md for the setsid/nohup differences.

Decision: do you need the child to be a session leader (independent of the parent's controlling TTY)?

• YES -> setsid cmd > log 2>&1 < /dev/null & -- creates a new session, detaches from the TTY.
• NO, just survive SIGHUP -> nohup cmd > log 2>&1 < /dev/null & -- ignores SIGHUP, inherits session.

Constraint: nohup prints nohup: ignoring input and appending output to 'nohup.out' if you do not redirect stdin/stdout/stderr. Redirect all three explicitly to keep logs where you control them -- because nohup.out in the working directory creates sprawl and may fail on read-only volumes.

Step 4: Resolve the real PID

Load references/pid-resolution.md -- this is the most-frequent failure class in the skill.

Problem: $! tells you the shell-level PID of the last backgrounded job. That is often a wrapper (nohup, time, stdbuf, env, sh -c '...') and not the process doing the work.

Resolve by querying the observable state:

Constraint: when you kill a wrapper PID, the wrapper dies and the orphaned child re-parents to PID 1 but keeps running. Always re-query after a kill -- the port still bound, the file still locked, pgrep still matches -- because assuming the kill worked is how production incidents start.

Gate: before declaring a process killed, run the same discovery query again and confirm it returns nothing -- see Step 6.

Step 5: Signal and trap discipline

Load references/signals-and-traps.md.

Key rules:

1. SIGTERM first, SIGKILL last. SIGTERM lets the process run its cleanup (flush buffers, release locks, delete PID files). SIGKILL is unmaskable and leaves lock files behind. Send SIGTERM, wait a bounded interval (typically 10s), then escalate.
2. Signal the process group, not the PID, when you want the whole tree. kill -TERM -$pgid (note the leading dash). Alternatively launch with setsid so the group ID equals the child's PID.
3. Subshells need their own trap setup. A trap set in a parent does not fire inside $(...) or (...). If the subshell has work to clean up, set the trap inside it.
4. exec cmd replaces the shell. All traps set before exec are gone -- because exec overlays the process image. If you need a wrapping shell with traps, do not exec; run the command as a child and wait for it.
5. trap 'handler' EXIT fires on normal exit, set -e exit, and most signals -- but not SIGKILL and not kill -9. Treat cleanup on SIGKILL as "impossible"; design the on-disk state to survive a hard kill.

Constraint: traps that call exit inside the handler can mask the real exit code. Use trap 'rc=$?; cleanup; exit $rc' EXIT to preserve it -- because losing the non-zero exit hides failures from CI.

Step 6: Verify cleanup

Load references/cleanup-verification.md.

Every destructive operation on a process ends with a verification query that asks the system "is it really gone?" -- not "did kill return 0?"

Pattern:

# 1. send SIGTERM
kill -TERM "$pid" 2>/dev/null || true

# 2. wait with a bound (condition-based-waiting covers the generic shape;
#    here we specialize to the kill-and-check shape)
for _ in {1..10}; do
    kill -0 "$pid" 2>/dev/null || break
    sleep 1
done

# 3. escalate if still alive
if kill -0 "$pid" 2>/dev/null; then
    kill -KILL "$pid" 2>/dev/null || true
    sleep 1
fi

# 4. verify the observable state, not just the PID
if kill -0 "$pid" 2>/dev/null; then
    echo "FATAL: $pid still alive after SIGKILL" >&2
    exit 1
fi
# Also re-query whatever resource this process was holding:
# ss -tlnp "sport = :8080" | grep -q . && echo "port still bound"

Constraint: the check must target the resource (port, lock file, device), not just the PID -- because a re-spawned process under a different PID can re-bind the port immediately, and "PID gone" is not the same as "resource free".

Gate: if the verification query still returns the resource as held, do not proceed. Surface the state, do not auto-escalate beyond SIGKILL -- because the next steps (reboot, fuser -k, kernel intervention) require a human decision.

Step 7: Strict-mode scripts

Default header for bash scripts that manage processes:

#!/usr/bin/env bash
set -euo pipefail

cleanup() {
    local rc=$?
    [[ -n "${child_pid:-}" ]] && kill -TERM "$child_pid" 2>/dev/null || true
    rm -f "${lock_file:-/dev/null}"
    exit "$rc"
}
trap cleanup EXIT
trap 'echo "ERROR: line $LINENO exit $?" >&2' ERR

Constraint: a command followed by || something has its exit code masked -- set -e does not fire, and neither does the ERR trap. Use || true only where failure is genuinely harmless, and prefer if ! cmd; then handle; fi when you want to act on the failure -- because silent-swallow || true is the top failure mode in this domain (see references/preferred-patterns.md).

Step 8: Verify

After implementing any pattern from Steps 2-7:

• Success path: child started, PID captured correctly, trap fires on normal exit.
• Failure path: child fails to start -> clear error, no orphan, no stale lock file.
• Kill path: SIGTERM received -> child exits cleanly; SIGKILL received -> no lock/PID file remains (if your design supports it) or the next run tolerates leftover state.
• Verification: port / lock / PID file is genuinely released, re-checked with ss / ls / kill -0.

Error Handling

Error: "kill: (1234): No such process" but the port is still bound

Cause: $! captured the PID of a wrapper (nohup, sh -c, time), not the real child. You killed the wrapper; the child re-parented to init and kept listening.

Solution: re-query with ss -tlnp "sport = :PORT" to get the real PID, then kill that. See references/pid-resolution.md. Fix the start-time code so the real PID is captured up front.

Error: trap handler never fires

Cause: one of (a) trap was set inside a subshell that already exited, (b) the script was replaced by exec, (c) the process received SIGKILL, (d) the trap was overwritten by a later trap 'other_handler' EXIT.

Solution: move the trap into the shell that actually owns the state. Do not exec if you need traps. Accept that SIGKILL bypasses traps and design the on-disk state to survive it.

Error: script hangs forever in wait

Cause: waiting on a PID that was already reaped, or on a PID that belongs to a different process group than the shell expects. Or SIGCHLD is being caught by a trap that never returns.

Solution: use wait -n (bash 4.3+) to wait for any child and return its exit code, or use a bounded loop with kill -0 checks. Audit SIGCHLD traps.

Error: set -e script "succeeds" but did not actually do the work

Cause: a command was followed by || true or || :, swallowing the real failure. Or the failing command was on the left side of a pipeline without pipefail.

Solution: remove || true unless you genuinely do not care. Add set -o pipefail. See references/preferred-patterns.md for the set -e + || swallowing pattern with detection command.

References

Loading Table

Reference Files

• references/starting-processes.md -- &, nohup, disown, setsid, daemonization, fd redirection.
• references/pid-resolution.md -- why $! lies, reliable PID capture, ss/pgrep/lsof recipes.
• references/signals-and-traps.md -- SIGTERM/SIGKILL, trap ordering, subshell inheritance, exec semantics, process groups.
• references/cleanup-verification.md -- kill-and-check pattern, state re-query after destructive operations.
• references/preferred-patterns.md -- concrete gotchas (nohup + $! wrapper PID, set -e + || swallowing, and more) with rg detection commands and paired fix snippets.