hipfire-autoheal
Runtime repair after evidence. Prefer diagnosis and the smallest reversible fix; do not kill processes, delete files, or reinstall until ownership and symptom match are confirmed.
For GPU/ROCm inventory and cold bring-up, start with hipfire-diag (or
hipfire diag). Use this skill when the install looks present but serve/run
is stuck, crashing, OOM, or semantically wrong.
Workflow
- Evidence first
.agents/skills/hipfire-autoheal/triage.sh
.agents/skills/hipfire-autoheal/triage.sh --json
hipfire ps
hipfire diag # optional fuller inventory
tail -n 80 ~/.hipfire/serve.log
- Map
likely_issues+ log lines throughplaybook.md(symptom → diagnosis → minimal repair). Do not skip earlier catalog rows without ruling them out. - Check
known-issues.mdfor current hardware/model caveats (dated / ref-pinned when available). Rows marked historical are fixed, superseded, retained measurement debt, or unverified older observations — do not treat them as live defaults without fresh evidence. - Use
bisection.mdonly when the catalog does not localize the failure. Daemon-side env bisections require a confirmed local path (see playbook / bisection guardrails) — prefixinghipfire runalone does not reconfigure a resident serve. - Verify the same user path that failed (serve health,
hipfire run, or the exact chat/completions request). Route any correctness/perf claim perdocs/VALIDATION.md— there is no universal gate.
Safe vs destructive actions
| Class | Examples | When |
|---|---|---|
| Safe automatic | Read-only triage/diag; hipfire ps; curl /health; one-shot env on a confirmed local daemon path (HIPFIRE_LOCAL=1 / no resident serve — see bisection); HIPFIRE_HIPCC_EXTRA_FLAGS; tail logs |
Always allowed while diagnosing |
| Safe controlled recovery | Plain hipfire stop only when it targets the tracked serve pid that passes CLI ownership validation; wait on cold JIT while tailing serve.log |
Prefer these before any force-reap path |
| Destructive / manual — need explicit user approval | hipfire restart -d (stop leg force-reaps name-matched daemons and fuser -k on the port — can kill an unvalidated port owner); hipfire stop --force / --all; scripts/serve-restart.sh (kills by pattern + fuser -k); bare pkill/kill -9; deleting serve.pid/daemon.pid outside CLI validation; wiping kernel cache; hipfire config set …; hipfire update; package installs; reboot; any privileged command |
Only after port/process-owner inspection (hipfire ps, pidfile, port listener, /health) and explicit user approval |
Never blind-kill or delete:
- Confirm the tracked serve via
hipfire ps, pidfile record, port owner, and/or/healthtoken before any kill path. Plainhipfire stoprefuses the tracked pid on failed ownership validation (crates/hipfire-cli/src/main.rs); that guarantee does not cover force-reap / restart port cleanup (fuser -k, exact-name daemon reap). Agents must inspect owners before requesting approval for restart/force. - Do not
rmJIT/kernel caches unless diagnosing a confirmed hash/arch mismatch. Cache root isHIPFIRE_KERNEL_CACHE(default cwd-relative.hipfire_kernels/<arch>/— seedocs/env-vars.md). - Do not reinstall ROCm/hipfire or rebuild the tree as a first step.
Canonical recovery paths
| Intent | Path |
|---|---|
| Tracked stop (ownership-validated pid only) | hipfire stop |
| Stop + orphan reap + free port | hipfire stop --force (approval + owner inspection first); optional --all also reaps quantize jobs |
| Stop then start with same flags | hipfire restart -d (approval — force-reap semantics on the stop leg; inspect port/process owners first) |
| Scripted kill/free/optional relaunch | scripts/serve-restart.sh [port] [--kill-only] [-- serve args…] — destructive; see script header + docs/SERVE.md |
| Read-only / diagnostic GPU/install inventory | hipfire diag or .agents/skills/hipfire-diag/ (GPU/JIT side effects disclosed there) |
| Serve API / lifecycle prose | docs/SERVE.md, docs/CLI.md |
| Config / env owners | docs/CONFIG.md, docs/env-vars.md |
| Claim → validation route | docs/VALIDATION.md |
Default bind: 0.0.0.0:11435. Pid/log: ~/.hipfire/serve.pid,
~/.hipfire/serve.log. Install root: ~/.hipfire/ (bin/daemon, models/,
config.toml).
References in this skill
| File | Role |
|---|---|
triage.sh |
Structured evidence gatherer (--json optional) |
playbook.md |
Symptom → diagnosis → repair catalog |
known-issues.md |
Dated current + historical/unknown caveats |
bisection.md |
Localize hangs/regressions after the catalog |
Out of scope
- Performance claims or route admissions (methodology +
VALIDATION.md/docs/admissions.ymlonly). - Blind coherence-gate runs:
scripts/coherence-gate-*.share historical reproduction only, not acceptance (docs/VALIDATION.md). - Editing product code unless the user asked for a code fix; this skill is runtime ops first.