Agentic Process Monitor

Agentic Process Monitor

Patterns for monitoring background processes from Claude Code — detecting success, failure, timeout, and hung processes, then returning results to the main context to drive the next action.

Companion skills: devops-tools:pueue-job-orchestration (remote/queued work) | devops-tools:distributed-job-safety (concurrency)

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

Architecture: Sentinel + Heartbeat + Agent

Main Context                          Monitor Agent (subagent)
─────────────                         ──────────────────────
1. Start work (Bash run_in_background)
   └─ work wrapper writes sentinel files
2. Launch Agent (poll every 15s) ────► poll loop:
3. Continue other work                  .status exists? → return result
                                        .heartbeat stale? → kill, return error
                                        elapsed > max? → kill, return timeout
4. Agent returns ◄──────────────────── detected outcome
5. Act on result (next step / retry / abort)

Why this architecture: The main context stays lean (no polling tokens burned). The subagent handles all the waiting. If the subagent itself fails, the main context can recover by checking sentinel files directly.

Sentinel Protocol

The work process writes 4 files to a known directory (e.g., /tmp/<project>-monitor/):

Work Wrapper Template

#!/usr/bin/env bash
set -uo pipefail
STEP="${1:?step name required}"
MONITOR_DIR="${2:-/tmp/monitor}"
mkdir -p "$MONITOR_DIR"

echo $$ > "${MONITOR_DIR}/${STEP}.pid"

# Heartbeat: touch file every 10s in background
(while true; do touch "${MONITOR_DIR}/${STEP}.heartbeat"; sleep 10; done) &
HB_PID=$!
trap "kill $HB_PID 2>/dev/null" EXIT

# === YOUR WORK HERE ===
if your_command --args 2>"${MONITOR_DIR}/${STEP}.log"; then
    echo "SUCCESS" > "${MONITOR_DIR}/${STEP}.status"
    echo '{"key": "value"}' > "${MONITOR_DIR}/${STEP}.result"
else
    echo "FAILED" > "${MONITOR_DIR}/${STEP}.status"
fi

Monitor Decision Tree

The polling agent checks every POLL_INTERVAL seconds (default: 15s):

every POLL_INTERVAL:
  .status exists?
    → read status + result, return to main context
  .heartbeat exists AND mtime stale (> STALE_THRESHOLD)?
    → process hung — kill PID, return "hung" error
  elapsed > MAX_TIMEOUT?
    → timeout — kill PID, return "timeout" error
  otherwise
    → sleep POLL_INTERVAL, continue

Recommended Defaults

Circuit Breaker

Prevent repeated failures from wasting compute. Three consecutive crashes or failures → stop and report. Reset counter on any success.

consecutive_failures = 0
MAX_CONSECUTIVE = 3

on failure:
  consecutive_failures += 1
  if consecutive_failures >= MAX_CONSECUTIVE:
    STOP — likely infrastructure, not the work itself

on success:
  consecutive_failures = 0

Agent Self-Healing

If the monitoring subagent fails (context overflow, timeout, crash), the main context recovers:

1. Check .status file directly — work may have finished while agent was dead
2. If .status exists → read result, continue normally
3. If no .status but .heartbeat is fresh → spawn replacement monitor agent
4. If no .status and .heartbeat is stale → process hung, kill PID, log error

This guarantees the main context never gets permanently stuck.

Anti-Patterns

Environment Preflight

Run before entering any autonomous loop. If any check fails, fix before proceeding.

# 1. Python package importable?
uv run --python 3.14 python -c "import your_package" \
  || uv sync --python 3.14 --extra dev

# 2. Console scripts have valid shebangs? (catches post-rename breakage)
uv run --python 3.14 pytest --co -q tests/ 2>/dev/null \
  || uv sync --python 3.14 --extra dev

# 3. External service reachable?
curl -sf "http://localhost:PORT/?query=SELECT+1" \
  || echo "FAIL: start service or SSH tunnel"

Common uv/venv Failures

Rule: After any repo rename, directory move, or Python version change → always uv sync --python 3.14 --extra dev before running anything.

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure.
2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match.
3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.