Joelclaw System Check

joelclaw System Health Check

Run scripts/postboot.sh for the read-only postboot/daily topology sweep. It checks the Flagg/Panda authority split, NAS mounts, NFS tuning, custom NAS MinIO, Central Postgres, local Convex, Typesense, Inngest, and shadow Central health without printing secrets. The daily automation also reviews recent Codex infra transcripts so completed work does not stay trapped in chat.

Migration caveat — 2026-07-09

The current health script is still too Panda-centric: it treats Panda-era k8s/PDS/NodePort surfaces as if they were the whole joelclaw system. That is no longer aligned with the Central migration vision.

Health needs to become lane-based:

• Central lane: Flagg/Mac Studio native launchd services, Redis, Typesense, Inngest, system-bus worker, agent-mail, run capture, session NAS backup receipts, reboot-survivable state.
• Relay lane: Panda-only leftovers such as iMessage/account-bound relays, legacy webhook/callback surfaces, and explicit decommission blockers.
• Satellite lane: Blaine/other Machines' JOELCLAW_CENTRAL_URL, outbox backlog, fresh capture timestamps, and transcript backup freshness.

Until that rewrite lands, read red Panda/k8s/PDS checks as migration TODOs, not proof that Flagg Central is dead.

~/Code/joelhooks/joelclaw-runtime/skills/joelclaw-system-check/scripts/postboot.sh

Run scripts/health.sh for the older full system health report with 1-10 score. This is heavier because it includes tests and TypeScript.

~/Code/joelhooks/joelclaw-runtime/skills/joelclaw-system-check/scripts/health.sh

Postboot/Daily Check

Use this after a reboot, after storage/network work, and for the daily automation.

Default command:

POSTBOOT_NAS_BENCHMARK_MIB=1 \
  ~/Code/joelhooks/joelclaw-runtime/skills/joelclaw-system-check/scripts/postboot.sh

Default posture:

• Read-only except for the tiny NAS verifier write probe under the already-designated s3 proof paths.
• Does not run the Convex write/read/delete smoke unless POSTBOOT_HEAVY=1.
• Does not run the legacy full health.sh unless POSTBOOT_RUN_FULL_HEALTH=1.
• Never prints raw .env, MinIO keys, Convex admin keys, or Postgres URLs with passwords.
• Treats custom NAS MinIO on 100.67.156.41:39000 as canonical.
• Treats ASUSTOR MinIO CE on 29990 as a warning/reference surface only.
• On Flagg, expects JOELCLAW_CENTRAL_URL and direct Typesense helpers to still point at Panda until explicit Central cutover.

Critical checks:

Warning/reference checks:

• custom MinIO console TCP on 39001
• ASUSTOR MinIO CE readiness on 29990
• aggregate joelclaw status and joelclaw inngest status CLI wrappers, because those can reveal local env drift even when the authoritative endpoint is healthy
• Convex LAN and tailnet forwards
• local Convex LAN forwarder LaunchAgent
• Flagg shadow Central health
• gated heavy checks

If postboot.sh fails, fix critical failures first. Do not chase warning/reference failures before the required path is green.

Recent Codex Transcript Review

The daily automation should review recent Codex infra sessions after the health script runs.

Preferred indexed search:

Use recall and the current session-search contract. Keep private queries in structured MCP arguments. Raw evidence requires a scope-bound evidenceDrilldownReceipt; missing projections do not grant raw access.

If indexed results are stale or missing, inspect metadata without reading transcript bodies. Raw content still requires the scoped receipt:

tail -n 80 ~/.codex/session_index.jsonl
find ~/.codex -type f -name "rollout-*.jsonl" -mtime -2

Review only bounded snippets, final answers, commands, and receipts. Do not dump full transcripts or secrets.

Promote durable facts into the right surface:

• system topology and Panda/Flagg authority -> skills/system-architecture/SKILL.md
• NAS access, LAN/MTU/NFS behavior -> skills/three-body/SKILL.md
• custom NAS MinIO behavior -> skills/minio/SKILL.md
• local Convex exposure and daily-use shape -> skills/local-convex/SKILL.md
• dated receipts and decisions -> .brain/resources/*.svx

Treat in-progress subagent threads and sessions without a final answer as leads, not architecture truth.

Active Codex automation:

• ID: daily-joelclaw-postboot-system-check
• Schedule: daily at 08:15 local time
• Workspace: /Users/joel/Code/joelhooks/joelclaw-runtime
• Command intent: run POSTBOOT_NAS_BENCHMARK_MIB=1 skills/joelclaw-system-check/scripts/postboot.sh, summarize failures/warnings, review recent Codex infra transcripts, report uncaptured durable facts, and avoid secret output.

Full Health Check (16 components)

Agent Session Capture + NAS Backup Check

Use the agent-session-capture-backup skill when the check needs proof that Pi/Claude/Codex transcript activity is captured and backed up across Flagg, Blaine, and Panda.

Fast audit/repair command from ~/Code/joelhooks/joelclaw:

bun scripts/agent-session-audit-backup.ts \
  --hosts flagg,blaine,panda \
  --central-url http://joels-mac-studio.tail7af24.ts.net:3111 \
  --backup-root /Volumes/three-body/sessions \
  --repair-env \
  --sync=true \
  --replay-outbox \
  --replay-limit 250 \
  --replay-max-bytes 10485760

Daily durable workflow:

• function: system/agent-session.capture-backup.verify
• event: system/agent-session.capture-backup.requested
• cron: TZ=America/Los_Angeles 15 5 * * *
• receipts: /Volumes/three-body/sessions/receipts/*.json

Treat stale or empty /Volumes/three-body/sessions as a critical backup failure even when runs_dev Typesense indexes are fresh. Typesense is derived; raw transcripts and run blobs are the source of truth.

When to Run

• Session start — orient on system state before doing work
• After loops complete — verify nothing broke
• After infra changes — k8s, worker, Redis config
• When something feels off — quick triage

Fixing Common Issues

Repo drift: cd ~/Code/joelhooks/joelclaw && git fetch origin && git status -sb

pi-tools broken: cd ~/.pi/agent/git/github.com/joelhooks/pi-tools && bun add @sinclair/typebox @mariozechner/pi-coding-agent @mariozechner/pi-tui @mariozechner/pi-ai

PDS unreachable: curl -fsS http://localhost:9627/xrpc/_health then kubectl get deploy,svc,pods,pvc -n joelclaw | rg 'bluesky-pds|NAME' (or if pod down: kubectl rollout restart deployment/bluesky-pds -n joelclaw)

Worker down: joelclaw inngest restart-worker --register

Failing tests: Inspect the affected behavior and repair valid coverage. Test provenance alone never authorizes removal.

System-bus test false reds: the health script runs each src/**/*.test.ts file in its own Bun process because several legacy tests monkey-patch globals or use mock.module. If the aggregate health check is green but raw bun test is red, suspect inter-file mock leakage before treating runtime code as broken.

Loop artifacts: Inspect task ownership and unharvested work. Remove only verified disposable artifacts within the authorized scope.

Inngest Hung-Run Quick Triage

When a run appears stuck after first step:

joelclaw run <run-id>

If trace shows Finalization failure with "Unable to reach SDK URL":

1. Verify registration/health: joelclaw inngest status

2. Verify function is present where expected: joelclaw functions | rg -i "manifest-archive|<function-name>"

3. Check for stale app registrations in Inngest UI/API and remove stale SDK URLs.

4. Assume possible handler blocking (not just network): review recent step code for filesystem/Redis/subprocess blocking before step response.