Mac Patrol

mac-patrol — the procedure

Engine: mac_patrol.py (single source of the logic — do not reimplement it here). This file is the ORDER OF OPERATIONS around the engine. Every step below was paid for on a live machine; the traps are annotated with what they actually cost.

Step 0 — "slow" is not one disease, and load average is not the evidence

Three different complaints, three different culprits. Jumping straight to zombies means fixing at random.

⚠️ Load average on a Mac with dozens of sessions means almost nothing. Measured: load 233 while only 4% of the CPU was busy — the processes were queued on MEMORY, not on the CPU. The honest pair of numbers is free CPU % from top plus swap used. Load is a before/after prop, not a diagnosis.

Step 1 — MEASURE (never kill blind)

top -l 1 -n 0 | head -12                  # load avg + memory
ps aux | awk '$8 ~ /^Z/ {print $2, $8}'   # zombies (state Z)
python3 mac_patrol.py --dry-run           # every decision, no action
memory_pressure | tail -3                 # the TRUTH about memory

⚠️ Traps, all paid for:

• "257 MB unused" in top does not mean you are out of memory. macOS deliberately keeps RAM busy as cache. The truth is memory_pressure → "System-wide memory free percentage" (70% = you are fine).
• ps %CPU is the average over the process's whole LIFETIME. A patrol can shout "STUCK AT 76%" about a process that went quiet hours ago. Before any kill:

  t1=$(ps -o time= -p <PID>); sleep 5; t2=$(ps -o time= -p <PID>); echo "$t1 -> $t2"
  

  Delta ≈ 0 over 5s = asleep, do NOT kill. Delta ≈ 5s over 5s = burning a whole core, now.
• ⚠️ One probe is not enough for a bursty daemon. Measured on searchpartyd inside a single session: probe #1 = 0.01s/5s (asleep — the accusation was false) → twenty minutes later probe #2 = 5.04s/5s (100% of a core — the accusation was true). Same process, opposite verdict. A verdict of "asleep" is good for minutes, not hours: measure immediately before the kill, not from memory. Borderline case = two or three probes spread out.

Step 1b — memory is eaten by COPIES of an MCP server, not by one fat process

Your watchdog complains about "a bloated MCP over 700 MB" — but the real eater is usually modest and numerous. The weight of a class is footprint × number of sessions:

for M in computer-use telegram-mcp claude-in-chrome; do
  printf "%-18s %2d procs  %5d MB\n" "$M" \
    $(pgrep -fc "$M") \
    $(( $(ps -Ao rss=,command= | grep "$M" | grep -v grep | awk '{s+=$1} END {print s+0}') / 1024 ))
done

Measured: one MCP server at 163 MB × 30 sessions = 4.9 GB, while every individual copy sat at a third of the "bloated" threshold, so nothing ever alarmed. Closing 13 sleeping sessions took it to 0.81 GB. That — not orphans, not zombies — was the actual cause of the slowdown.

The class-level fix is one shared daemon per machine instead of one child per session. We did it for one MCP server (SSE daemon on a fixed port): 48 copies × 378 MB = 11.7 GB became 4 copies / 97 MB. Where no such diet exists yet, closing idle sessions is the only lever you have.

Step 2 — zombies: first check whether they are GROWING (usually they are not)

⚠️ Measure before you kill. There are two entirely different kinds of zombie, and the cure is opposite.

Z1=$(ps -Ao state|grep -c Z); sleep 45; Z2=$(ps -Ao state|grep -c Z); echo "$Z1 -> $Z2"
ps -Ao pid,ppid,etime,state,ucomm | awk '$4 ~ /^Z/'   # is their age == the parent's age?

• Count is flat, zombie age == parent age → not a leak, but a permanent state of some program. Do nothing, report it in one line.
• Count grows between probes → that is a leak. Go to Step 2b.

Measured: 2 → 2 over 45 seconds, four mentions in two days of logs. Verdict: not a leak. Killing a useful service's parent over two harmless zombies is a bad trade.

Step 2b — if they really grow: kill the PARENT, never the zombie

A zombie (STAT=Z, <defunct>) is already dead — kill on it does nothing. It lingers because its parent never collected the exit code. Find the parent:

ps -o pid,ppid,stat,command -p <ZPID>
ps -o pid,command -p <PPID>

⛔ First rule: the kill target is the zombie's PPID, and it must never be 1. PID 1 is launchd; killing it is a kernel panic. If the zombie's PPID is already 1, it has been adopted by launchd and there is nothing to do at all — launchd will reap it. Gate this before any kill:

PPID=$(ps -o ppid= -p <ZPID> | tr -d ' ')
[ "$PPID" = "1" ] && { echo "adopted by launchd — leave it"; exit 0; }
[ -z "$PPID" ]    && { echo "parent already gone"; exit 0; }

Then branch on what the parent is (below, <PPID> is always the zombie's parent, and ≠ 1):

• The parent is itself an orphan (the PARENT's own ppid is 1) — e.g. an abandoned geckodriver from a dead Selenium run → kill <PPID> freely; the zombie moves to launchd and gets reaped. Check the ppid of the PARENT, not of the zombie: ps -o ppid= -p $PPID.
• The parent is a well-behaved app that simply never reaps → ⛔ leave it alone; this is a false target. Measured on a desktop time-tracker: killed the watcher → restarted it → the new watcher spawned exactly the same pair of zombies within two minutes. It does not reap its two children, ever; that is its permanent state, not a leak. kill→restart is whack-a-mole: the counter resets, the root stays. Cost of the zombies: two rows in the process table, zero CPU, zero memory. The real fix is upstream (update or disable the app), and that is the human's call. ⚠️ Also: some apps do NOT respawn their own helper after you kill it. If you killed it, restart it by hand.
• The parent is a live working session (your shell, your editor, your agent) → do not kill. The zombie is harmless. Report and move on.

Step 3 — CPU hogs and hung LLM calls

• A hung headless LLM call (gemini -p, claude -p, codex running > 15 minutes): walk ps -o ppid= upward to find the owner. If the owner is a live session waiting on that call, killing the CALL is the RIGHT move — the session unblocks and regains control. ⚠️ Name the tree explicitly, and guard against PID reuse. The target is the call plus its wrapper (for gemini that is a PAIR of node processes: node .../gemini and its child node --max-old-space-size=…), but ⛔ never walk up as far as the owning session — that one must survive. A PID can be freed and reused between your ps and your kill, so re-check the target immediately before killing it:

  for P in <PID_wrapper> <PID_call>; do
    ps -o command= -p "$P" | grep -q 'gemini' && kill "$P" || echo "PID $P is a different process now — skip"
  done
  

  Take the age from the CALL (ps -o etime= -p <PID>), not from the owning session.

• Root daemons (searchpartyd, mds, …): cannot be killed without sudo, and this procedure never types a password. Delta-measure first (Step 1) — most of the time it has already gone quiet. Genuinely stuck → hand the human one line: sudo kill <PID> (launchd will restart it).

• A GUI renderer — branch on AGE, not on percentage. A fresh renderer at 100% is a live window doing work: leave it. A renderer that is burning and older than 12 hours is stuck: kill it, the app repaints and your data lives server-side. Measured: one renderer burned a core for 4 days 16 hours (91 minutes of pure CPU time) with load average at 37; killing it took the load to 18. The robot does this itself (RESTARTABLE_HOG + STUCK_AGE_H=12), so you should not need to.

• ⛔⛔ System processes a naive patrol will call "stuck" but which must NOT be killed. A 12-hour threshold is meaningless for daemons that are SUPPOSED to live the whole session:

  Process	A naive patrol says	What a kill actually does
  WindowServer	"STUCK 56h, 82% cpu"	logs you out; everything unsaved is gone
  launchd (pid 1)	—	kernel panic
  fileproviderd	"108% cpu"	tears a cloud-drive sync in half
  your sync daemon	"hog"	stops file delivery you depend on

  Their burning is a SYMPTOM of load you created (many windows, many sessions, many files in flight); the cure is removing that load, not the kill. Measured: WindowServer burning 77% of a core — the root cause was 36 simultaneous agent sessions, not WindowServer.

• ⛔⛔ An unfamiliar name at the top of top — ATTRIBUTE FIRST, judge second. Identification takes a minute and answers the only question that matters: whose work is this?

  ps -o pid,ppid,etime,command= -p <PID>            # the full command line
  ls -l  $(which <name> || echo /path/to/bin)       # WHEN was it installed = who could have
  file   /path/to/bin ; strings /path/to/bin | head -50
  P=<PID>; while [ "$P" != 1 ]; do ps -o ppid=,command= -p $P; P=$(ps -o ppid= -p $P|tr -d ' '); done
  

  Measured: an unknown binary burning 728% CPU and 1.05 GB looked like a runaway. One minute of identification: installed that same morning, an MCP server and repository indexer inside, and the ppid chain led to a live agent session in the next window that had run it deliberately. Nine minutes later it exited on its own. Killing it would have severed someone's work blindly. Rule: live work is not killed on the strength of its position in top. Owner first, verdict second. Could not attribute it? Report, do not kill.

• ⛔ Never kill: live sessions younger than 24h; anything whose burning is not confirmed by a DELTA.

Step 3b — after killing a Spotlight daemon, expect a loud reindex. That is correct.

The indexer comes back, and for the first few minutes it works harder than the stuck one did:

kill corespotlightd + spotlightknowledged   (stuck for 128 hours)
   → a minute later: mds_stores 47%, installd 34%, fresh corespotlight daemons
   → 10-20 minutes later: quiet

Two consequences, both mandatory:

• Do not take the before/after verdict immediately, take it after things settle. Otherwise honest work looks like a failure: "I killed the hogs and it got worse" — those are the same daemons, now doing their job.
• ⛔ Do not kill the reindex. What saves you from an infinite kill-loop is the SYSTEM_STUCK_H = 2 threshold: fresh daemons are immune by design. Do not reach past it by hand.

Step 4 — VERIFY (a kill is silent; count the survivors)

zsh does not word-split $VAR in a for loop, so a kill loop can quietly become a no-op:

ps -p <PID1>,<PID2>,... | tail -n +2   # empty = all dead
ps -Ao state | grep -c '^Z'            # compare against the BASELINE, not against zero
uptime                                  # did load average fall?

⚠️ The zombie criterion is the BASELINE, not zero. Zero is unreachable and, as a criterion, false: a permanent pair of zombies from some app (Step 2) will fail every honest cleanup you ever run.

# before:     BASE=$(ps -Ao state | grep -c '^Z')
# after:      AFTER=$(ps -Ao state | grep -c '^Z')    # expect AFTER <= BASE
# not growing: sleep 45; ps -Ao state | grep -c '^Z'  # third number == AFTER

Growing between probes → that is a leak, go to Step 2. Flat → you are done.

Report as before → after: load average, zombie count, what was killed and why, and — equally important — what was NOT touched and why. The reason for every kill must be proved by a delta or by a parentage chain, never by "it looked wrong".

What the robot already does, so you do not do it by hand

• Measures burning by DELTA of two CPU-time snapshots on every tick, never by ps %CPU.
• Class "STUCK": burning > 40% of a core AND older than 12h. Restartable app renderers get restarted; anything else is reported with a "STUCK Nh" marker.
• Counts concurrent sessions: more than 20 → an alarm line, because each one drags its own MCP servers. This, not orphans, is the main memory cost on a busy machine.
• Closes sessions idle for more than 5 hours. Key measurement: a sleeping session still ticks at ~4-5% CPU (its own event loop), so the "asleep" threshold is <10% by honest delta; a genuinely working one burns 30-100%+ and lives as long as it needs. Without a delta measure (Windows) only the hard 24h ceiling applies, so a night robot never gets shot. Sessions are resumable from transcripts — closing one loses nothing.
• Reports bloated MCP servers (>700 MB) rather than killing them: many hang directly off a live app, and a blind kill severs a live chat.
• Takes its node name from a stable source, not from environment variables. Root cause we paid for: under launchd the environment is EMPTY, so the node named itself differently in scheduled runs than in interactive ones and wrote a SECOND state file — which then went stale silently under a freshness watchdog that was watching the other one. ⚠️ When you patch anything a scheduler runs, test it in a bare environment (env -i HOME=$HOME …), or your fix rests on a variable the robot does not have.

The system tier — cleaning up OS processes too

"System" is decided by PATH, not by a list of names. A name list is always incomplete: in the first version airportd went unrecognised purely because its name was missing. Path marks (/System/Library, /usr/libexec/, /Library/Apple/, /usr/sbin/) cannot be faked.

Three buckets, and the border is the process OWNER:

Its own threshold: SYSTEM_STUCK_H = 2 hours, not the 12 used for a GUI renderer. A system daemon that has burned a core for two hours is not busy — it is in a loop.

⚠️ The trap that cost us the most. We announced that the system antivirus had been stuck for five days grinding the CPU, taking the number from ps %CPU = 91.4%. The delta probe: 10:10.83 → 10:10.83 over five seconds — the process was asleep, and the diagnosis was 100% wrong. That is precisely the trap Step 1 of this very document warns about, and we walked into it anyway. Before saying anything about a system process, take the delta — and do not even look at ps %CPU. The system tier therefore judges only by cpu_delta_map; on Windows, where the delta is not implemented, the tier does not fire at all (honestly empty, rather than a fake zero).

Tier tests: 11 checks inside test_mac_patrol.py (54/54 green) — our own daemon gets fixed, root is not touched and comes back as a line, WindowServer survives even when we own it, a young daemon is left alone, and the allowlist outranks the tier.

Related

• Engine: mac_patrol.py — --dry-run reports only, --verify exits 0 iff the schedule is alive and ran within 2 hours.
• Allowlist: mac_patrol_allow.txt — your own long-lived daemons look exactly like orphans (PPID=1). List them there before the first scheduled run.