Host Doctor

Host Doctor Workflow

CONCEPT:INFRA-001

Onboard-diagnose a host (or fleet) — OS validation + resource utilization + process / zombie / runaway / swap inspection + physical-storage/BMC health — and remediate only under explicit per-host approval. The four diagnostic steps are read-only and run in parallel; remediation delegates to systems-issue-troubleshooter and is gated behind a user decision (drive faults are surfaced as hardware replace candidates, never auto-remediated).

Steps

Step 0: hardware_profile_sweep

Agent: discovery-agent Tools: tun_tm_system, tun_tm_hosts

Run the hardware-profile-sweep atomic skill to validate each target's OS/kernel/arch, CPU, RAM, disks, and any GPU/accelerator. Establishes the baseline the load numbers are judged against (load must be read relative to core count). Expected: os, cores, accelerator

Step 1: host_resource_sampler

Agent: discovery-agent Tools: tun_tm_system, tun_tm_hosts

Run the host-resource-sampler atomic skill to sample CPU%, load average, memory %, and disk % across the target host(s), flagging any node over the utilization thresholds. Expected: cpu, load, memory, disk

Step 2: host_process_inspector

Agent: inspector-agent Tools: tun_tm_remote, sm_process_operations

Run the host-process-inspector atomic skill to rank CPU/RAM offenders by full cmdline, flag orphaned-to-PID1 runaways, list zombie→parent pairs, detect swap exhaustion + top swap holders and D-state blockers, and attribute culprit PIDs to their tmux session where possible. Produces the prioritized kill/reap/tune candidate list. Expected: runaways, zombies, swap, candidates

Step 3: storage_bmc_health

Agent: inspector-agent Tools: sm_storage_health, fan_manager_sel, fan_manager_sensors

Run sm_storage_health (action report) to check physical disk health that the filesystem/process steps miss: SMART for every drive incl. RAID megaraid passthrough, in-band PERC/LSI PD state, and BMC/IPMI drive-slot faults — correlated, so a BMC-flagged disk with clean SMART media is classified as a link/aging fault (reseat/replace), not media wear. For a remote host pass host=<inventory-key>; for out-of-band BMC pass params_json={"oob":true} (the iDRAC credential is read from OpenBao apps/idrac at runtime). For deeper BMC detail use the fan-manager IPMI tools (fan_manager_sel for the System Event Log history, fan_manager_sensors action type sensor_type Drive Slot). Flags failed/predicted-fail/faulted drives as replace candidates (hardware action — never auto-remediated). Expected: disks, bmc_drive_faults, faults

Step 4: present_and_gate [depends_on: 0, 1, 2, 3]

Agent: inspector-agent Tools: tun_tm_remote

Present the consolidated dashboard (OS validation + utilization + the ranked hog/zombie/ swap findings + drive/BMC faults + load-vs-cores verdict). Ask the user which candidates to act on. Drive faults are surfaced as hardware replace candidates (physical action, not auto-remediated). No change proceeds without an explicit, per-host choice. Safest-first ordering: orphaned runaways → reclaim swap → reap zombies (via parents) → wind down stale sessions.

Step 5: remediate [depends_on: 4]

Agent: remediation-agent Tools: sm_process_operations, sm_system_operations

Execute only the approved actions by delegating to the systems-issue-troubleshooter workflow (kill_process for confirmed rogue/orphan PIDs, clean_temp_files / clean_package_cache, swap reset). Verify each PID's identity (re-read cmdline) before killing — PIDs recycle. Never kill a process still owned by a live interactive session without naming the session and getting per-session confirmation.

Step 6: verify_and_persist [depends_on: 5]

Agent: discovery-agent Tools: tun_tm_system, graph_write

Re-run host-resource-sampler to confirm the load/swap recovered (load decays over ~1–15 min — re-sample, don't expect an instant drop). Persist the run (findings, actions taken, before/after metrics) to the Knowledge Graph via graph_write.

Output

• Host Doctor results persisted in KG (:HardwareNode health + remediation audit)
• Structured report (MD/PDF): OS validation, utilization, ranked hogs/zombies/swap, actions
• Audit trail with timestamps, the PIDs acted on, and per-host approvals

Execution

Run this workflow as a dependency-ordered DAG. Steps with no unmet depends_on run in parallel; dependents run after their prerequisites complete.

• Run first (in parallel): Step 0 — hardware_profile_sweep; Step 1 — host_resource_sampler; Step 2 — host_process_inspector; Step 3 — storage_bmc_health
• After level 0: Step 4 — present_and_gate (user decision)
• After Step 4: Step 5 — remediate (only approved actions, via systems-issue-troubleshooter)
• After Step 5: Step 6 — verify_and_persist

Execution: If graph-os is reachable, offload the whole DAG via graph_orchestrate action=execute_workflow (or the kg-delegate skill) for true parallel/swarm execution. Otherwise execute the steps natively in dependency order: run steps with no unmet depends_on in parallel, then their dependents.