Fleet Health

Fleet Health

Fast, read-only health snapshot across all 5 machines in the personal fleet. Unlike spark-audit / jetson-audit (deep, single-machine, best-practices audits), this skill is a wide, shallow, fleet-wide pulse check — designed to answer "is everything up?" in well under a minute, with a final line a headless cron job can grep for.

This skill reads the live system. It never modifies it.

No paths: activation is configured — this skill runs only on explicit invocation ("fleet status", "morning brief", "is the spark available", etc.). Since it never auto-triggers, no loop guard is needed.

Fleet Config

ssh:
  key: "~/.ssh/id_claude_code"
  opts: "-o ConnectTimeout=5 -o BatchMode=yes"
  user: "claude"

curl:
  opts: "-sf -m 5"

fleet:
  - name: "DGX Spark"
    host: "spark.k4jda.net"
    lan_ip: "192.168.10.33"
    role: "LLM inference (vLLM) — chat, embeddings, NER"
    gpu: nvidia-smi
    key_ports:
      8000: { service: "spark-llm (Qwen3.6-35B-A3B-FP8)", check: "GET /v1/models, expect data[0].id == spark-llm" }
      8001: { service: "qwen3-embedding-4b",               check: "GET /v1/models, expect data[0].id == qwen3-embedding-4b" }
      8002: { service: "GLiNER NER",                       check: "GET /health, expect status == ok" }

  - name: "Jetson Orin Nano"
    host: "jetson.k4jda.net"
    lan_ip: "192.168.10.58"
    role: "Edge LLM inference (llama.cpp), single-process mode-switched"
    gpu: tegrastats   # NO nvidia-smi on this box — do not attempt it
    key_ports:
      8080: { service: "active llama-server mode (default qwen35)", check: "GET /health, expect status == ok" }
      8081: { service: "embedding mode (Qwen3-Embedding-4B)",        check: "see Known States — U2: expected DOWN unless mode.txt=embedding" }

  - name: "Homeserver"
    host: "homeserver.k4jda.net"
    role: "Docker host (Unraid) — ~55 containers, Prometheus/Grafana/Alertmanager stack"
    sudo_scoped: true   # `sudo -n docker ...` works; `sudo cat`/`sudo ls` do NOT — use plain cat/ls or docker's own output
    use_wget_locally: true  # curl is not reliably available ON the homeserver itself — use wget when SSH'd in

  - name: "Bond"
    host: "bond.k4jda.net"
    role: "General purpose"

  - name: "open-brain-vm (obvm)"
    host: "obvm.k4jda.net"
    role: "T2 Claude CLI batch, Python, ops"

Execution

Run all checks in parallel — background each machine's SSH session and each spark/jetson curl probe with &, then wait. Target: under 60 seconds total. Do not run hosts sequentially.

Universal checks (all 5 hosts)

For each host, over one SSH session:

ssh -i ~/.ssh/id_claude_code -o ConnectTimeout=5 -o BatchMode=yes claude@<host>.k4jda.net \
  "uptime; df -h / ; df -h --output=pcent,target 2>/dev/null | sort -rh | head -5; free -h"

Collect: uptime + 1/5/15-min load, / usage, the single largest non-/ mount, and memory (total/used/available).

Spark-specific

curl -sf -m 5 http://192.168.10.33:8000/v1/models   # expect data[0].id == "spark-llm"
curl -sf -m 5 http://192.168.10.33:8001/v1/models   # expect data[0].id == "qwen3-embedding-4b"
curl -sf -m 5 http://192.168.10.33:8002/health      # expect {"status":"ok",...}
ssh ... claude@spark.k4jda.net "nvidia-smi --query-gpu=temperature.gpu,utilization.gpu --format=csv,noheader"

Jetson-specific

curl -sf -m 5 http://192.168.10.58:8080/health      # expect {"status":"ok"}
ssh ... claude@jetson.k4jda.net "timeout 3 tegrastats --interval 1000 | head -1"
# :8081 — evaluate per Known States (U2) below, do NOT treat a bare connection-refused as an anomaly on its own
curl -sf -m 5 http://192.168.10.58:8081/health

No nvidia-smi on the Jetson — always use tegrastats for GPU/thermal reads there.

Homeserver-specific

ssh ... claude@homeserver.k4jda.net "sudo -n docker ps --format '{{.Names}} {{.Status}}'"
# count containers whose status contains "unhealthy" or "(restarting"
ssh ... claude@homeserver.k4jda.net "df -h /mnt/user | tail -1"
# Prometheus is loopback-bound on the homeserver — NEVER call it directly from outside.
# Reach it only via SSH, and use wget (not curl) once you're on the box itself:
ssh ... claude@homeserver.k4jda.net "wget -qO- http://127.0.0.1:9090/api/v1/alerts"

Sudo on the homeserver is command-scoped: sudo -n docker ... works, but sudo cat/sudo ls do not — don't route file reads through sudo.

Known States — False-Alarm Prevention

Encode these before classifying anything as an anomaly. Each was verified against live behavior, not guessed.

U2 — Jetson :8081 embedding server (RESOLVED, expected-down). The Jetson runs a single llama-server process at a time, selected by ~/llm-server/mode.txt (qwen35 [default], nemotron, experiment, llm — all port 8080 — or embedding, port 8081). Modes are mutually exclusive; only one port is ever live. Production default is qwen35, so port 8081 is normally unreachable — this is by design, not a fault. Verified 2026-07-12: :8080/health → {"status":"ok"}, :8081 → connection refused. Treat :8081 connection-refused as expected-down, not an anomaly, as long as :8080/health is healthy. Only escalate 8081 if :8080 is ALSO down (that would indicate the whole llama-server process, not just the mode, is gone), or if mode.txt is known to have been switched to embedding and it's still unreachable.

Homeserver load average. This box runs 55 containers on modest core count; a resting load average around **3.5 is normal**, not degraded. Load also spikes transiently (observed 1-min spikes near 9 during bursts of parallel SSH/docker activity) while the 5/15-min trailing averages stay near baseline. Judge load health on the 15-minute average, not the 1-minute figure — a high 1-min with a normal 15-min average is noise, not an anomaly.

Nightly AppdataBackup window (~00:00–03:00 EDT). The Unraid AppdataBackup plugin stops and restarts containers during this window as part of a consistent backup. A container observed with a fresh StartedAt but an unchanged Created timestamp and RestartCount: 0 during or shortly after this window is a normal backup-cycle restart, not a crash. Only flag it if RestartCount > 0 or the restart falls outside the backup window.

Jetson thermal reads — millidegree gotcha. Raw thermal_zone*/temp files (and any node-exporter/textfile-collector wired directly to them) report millidegrees C, not degrees — a raw value like 51906 means 51.9°C, not a runaway 51,906°C thermal event. This skill sidesteps the issue by reading tegrastats (already human-readable, e.g. cpu@51.468C) rather than the raw thermal_zone files — but if this check is ever extended to read /sys/devices/virtual/thermal/thermal_zone*/temp directly, divide by 1000 before comparing against any threshold.

Anomaly Thresholds

Output Format

Render exactly one table, then an optional ANOMALIES section, then exactly one verdict line.

| machine | up | disk | key service | anomaly |
|---------|----|----|-------------|---------|
| DGX Spark | up 27d | 42% (2.0T free) | 8000/8001/8002 OK, GPU 41C/0% | — |
| Jetson Orin Nano | up 12d | 17% | 8080 OK; 8081 down (expected, U2) | — |
| Homeserver | up 93d | /mnt/user 31% (21T free) | 0 unhealthy/restarting of 58, alerts: none | — |
| Bond | up 10d | 11% | n/a (general purpose) | — |
| obvm | up 90d | 46% | n/a (general purpose) | — |

VERDICT: ALL HEALTHY

Rules:

• Include the ANOMALIES section only when at least one row has a non-— anomaly. Omit it entirely on a clean run — don't print an empty section.
• The final line of output is always either VERDICT: ALL HEALTHY or VERDICT: ACTION NEEDED — <short reason>. This must be the literal last line, unindented, so a headless cron job can grep '^VERDICT:' reliably.
• Keep the table to the 5 required columns — don't add columns per-machine; put machine-specific detail (model IDs, GPU temps, container counts) inside the key service and anomaly cells.

Error Handling

• Any host's SSH or curl probe exceeds its timeout (ConnectTimeout=5, curl -m 5): mark that host/service unreachable and count it as an anomaly (see Anomaly Thresholds) — except Jetson :8081, which is expected-down per Known State U2.
• A host is fully unreachable (SSH fails entirely): still render its row with up: unreachable and roll it into VERDICT: ACTION NEEDED rather than omitting the row.
• nvidia-smi invoked against the Jetson: never do this — use tegrastats only (see Jetson-specific checks); treat an accidental nvidia-smi failure there as a script bug, not a host anomaly.
• Homeserver sudo cat/sudo ls invoked: these fail because sudo is command-scoped to docker/nvidia-smi/systemctl — use plain cat/ls or docker's own output instead (see Homeserver-specific checks).
• Prometheus check fails when called directly from outside the homeserver: expected — it's loopback-bound; only reach it via SSH + wget from on-box (see Homeserver-specific checks), not a fault to report.