hipfire-autoheal
Runtime repair after evidence. Prefer diagnosis and the smallest reversible
fix; do not kill processes, delete files, or reinstall until ownership and
symptom match are confirmed.
For GPU/ROCm inventory and cold bring-up, start with hipfire-diag (or
hipfire diag). Use this skill when the install looks present but serve/run
is stuck, crashing, OOM, or semantically wrong.
Workflow
- Evidence first
.agents/skills/hipfire-autoheal/triage.sh
.agents/skills/hipfire-autoheal/triage.sh --json
hipfire ps
hipfire diag # optional fuller inventory
tail -n 80 ~/.hipfire/serve.log
- Map
likely_issues + log lines through playbook.md (symptom → diagnosis
→ minimal repair). Do not skip earlier catalog rows without ruling them out.
- Check
known-issues.md for current hardware/model caveats (dated /
ref-pinned when available). Rows marked historical are fixed,
superseded, retained measurement debt, or unverified older observations —
do not treat them as live defaults without fresh evidence.
- Use
bisection.md only when the catalog does not localize the failure.
Daemon-side env bisections require a confirmed local path (see playbook /
bisection guardrails) — prefixing hipfire run alone does not reconfigure
a resident serve.
- Verify the same user path that failed (serve health,
hipfire run, or
the exact chat/completions request). Route any correctness/perf claim per
docs/VALIDATION.md — there is no universal
gate.
Safe vs destructive actions
| Class |
Examples |
When |
| Safe automatic |
Read-only triage/diag; hipfire ps; curl /health; one-shot env on a confirmed local daemon path (HIPFIRE_LOCAL=1 / no resident serve — see bisection); HIPFIRE_HIPCC_EXTRA_FLAGS; tail logs |
Always allowed while diagnosing |
| Safe controlled recovery |
Plain hipfire stop only when it targets the tracked serve pid that passes CLI ownership validation; wait on cold JIT while tailing serve.log |
Prefer these before any force-reap path |
| Destructive / manual — need explicit user approval |
hipfire restart -d (stop leg force-reaps name-matched daemons and fuser -k on the port — can kill an unvalidated port owner); hipfire stop --force / --all; scripts/serve-restart.sh (kills by pattern + fuser -k); bare pkill/kill -9; deleting serve.pid/daemon.pid outside CLI validation; wiping kernel cache; hipfire config set …; hipfire update; package installs; reboot; any privileged command |
Only after port/process-owner inspection (hipfire ps, pidfile, port listener, /health) and explicit user approval |
Never blind-kill or delete:
- Confirm the tracked serve via
hipfire ps, pidfile record, port owner, and/or
/health token before any kill path. Plain hipfire stop refuses the
tracked pid on failed ownership validation (crates/hipfire-cli/src/main.rs);
that guarantee does not cover force-reap / restart port cleanup
(fuser -k, exact-name daemon reap). Agents must inspect owners before
requesting approval for restart/force.
- Do not
rm JIT/kernel caches unless diagnosing a confirmed hash/arch
mismatch. Cache root is HIPFIRE_KERNEL_CACHE (default cwd-relative
.hipfire_kernels/<arch>/ — see docs/env-vars.md).
- Do not reinstall ROCm/hipfire or rebuild the tree as a first step.
Canonical recovery paths
| Intent |
Path |
| Tracked stop (ownership-validated pid only) |
hipfire stop |
| Stop + orphan reap + free port |
hipfire stop --force (approval + owner inspection first); optional --all also reaps quantize jobs |
| Stop then start with same flags |
hipfire restart -d (approval — force-reap semantics on the stop leg; inspect port/process owners first) |
| Scripted kill/free/optional relaunch |
scripts/serve-restart.sh [port] [--kill-only] [-- serve args…] — destructive; see script header + docs/SERVE.md |
| Read-only / diagnostic GPU/install inventory |
hipfire diag or .agents/skills/hipfire-diag/ (GPU/JIT side effects disclosed there) |
| Serve API / lifecycle prose |
docs/SERVE.md, docs/CLI.md |
| Config / env owners |
docs/CONFIG.md, docs/env-vars.md |
| Claim → validation route |
docs/VALIDATION.md |
Default bind: 0.0.0.0:11435. Pid/log: ~/.hipfire/serve.pid,
~/.hipfire/serve.log. Install root: ~/.hipfire/ (bin/daemon, models/,
config.toml).
References in this skill
| File |
Role |
triage.sh |
Structured evidence gatherer (--json optional) |
playbook.md |
Symptom → diagnosis → repair catalog |
known-issues.md |
Dated current + historical/unknown caveats |
bisection.md |
Localize hangs/regressions after the catalog |
Out of scope
- Performance claims or route admissions (methodology +
VALIDATION.md /
docs/admissions.yml only).
- Blind coherence-gate runs:
scripts/coherence-gate-*.sh are historical
reproduction only, not acceptance (docs/VALIDATION.md).
- Editing product code unless the user asked for a code fix; this skill is
runtime ops first.
1---2name: hipfire-autoheal3description: Triage and repair hipfire runtime failures such as daemon hangs, stale serve.pid, port 11435 conflicts, ROCm include-path problems, missing precompiled kernels, VRAM OOM, kernel JIT failures, and multi-turn recall regressions. Use after diagnostics identify a likely runtime issue or when the user asks to fix a broken hipfire serve/run flow.4---56# hipfire-autoheal78Runtime repair after evidence. Prefer diagnosis and the smallest reversible9fix; do not kill processes, delete files, or reinstall until ownership and10symptom match are confirmed.1112For GPU/ROCm inventory and cold bring-up, start with `hipfire-diag` (or13`hipfire diag`). Use this skill when the install looks present but serve/run14is stuck, crashing, OOM, or semantically wrong.1516## Workflow17181. **Evidence first**1920```bash21.agents/skills/hipfire-autoheal/triage.sh22.agents/skills/hipfire-autoheal/triage.sh --json23hipfire ps24hipfire diag # optional fuller inventory25tail -n 80 ~/.hipfire/serve.log26```27282. Map `likely_issues` + log lines through `playbook.md` (symptom → diagnosis29 → minimal repair). Do not skip earlier catalog rows without ruling them out.303. Check `known-issues.md` for **current** hardware/model caveats (dated /31 ref-pinned when available). Rows marked **historical** are fixed,32 superseded, retained measurement debt, or unverified older observations —33 do not treat them as live defaults without fresh evidence.344. Use `bisection.md` only when the catalog does not localize the failure.35 Daemon-side env bisections require a confirmed local path (see playbook /36 bisection guardrails) — prefixing `hipfire run` alone does not reconfigure37 a resident serve.385. Verify the **same user path** that failed (serve health, `hipfire run`, or39 the exact chat/completions request). Route any correctness/perf claim per40 [`docs/VALIDATION.md`](../../../docs/VALIDATION.md) — there is no universal41 gate.4243## Safe vs destructive actions4445| Class | Examples | When |46|---|---|---|47| **Safe automatic** | Read-only triage/diag; `hipfire ps`; `curl` `/health`; one-shot env on a **confirmed local** daemon path (`HIPFIRE_LOCAL=1` / no resident serve — see bisection); `HIPFIRE_HIPCC_EXTRA_FLAGS`; `tail` logs | Always allowed while diagnosing |48| **Safe controlled recovery** | Plain `hipfire stop` **only when** it targets the tracked serve pid that passes CLI ownership validation; wait on cold JIT while tailing `serve.log` | Prefer these before any force-reap path |49| **Destructive / manual — need explicit user approval** | `hipfire restart -d` (stop leg force-reaps name-matched daemons and `fuser -k` on the port — can kill an unvalidated port owner); `hipfire stop --force` / `--all`; `scripts/serve-restart.sh` (kills by pattern + `fuser -k`); bare `pkill`/`kill -9`; deleting `serve.pid`/`daemon.pid` outside CLI validation; wiping kernel cache; `hipfire config set …`; `hipfire update`; package installs; reboot; any privileged command | Only after **port/process-owner inspection** (`hipfire ps`, pidfile, port listener, `/health`) and explicit user approval |5051Never blind-kill or delete:5253- Confirm the tracked serve via `hipfire ps`, pidfile record, port owner, and/or54 `/health` token before any kill path. Plain `hipfire stop` refuses the55 **tracked pid** on failed ownership validation (`crates/hipfire-cli/src/main.rs`);56 that guarantee does **not** cover force-reap / restart port cleanup57 (`fuser -k`, exact-name daemon reap). Agents must inspect owners before58 requesting approval for restart/force.59- Do not `rm` JIT/kernel caches unless diagnosing a confirmed hash/arch60 mismatch. Cache root is `HIPFIRE_KERNEL_CACHE` (default cwd-relative61 `.hipfire_kernels/<arch>/` — see [`docs/env-vars.md`](../../../docs/env-vars.md)).62- Do not reinstall ROCm/hipfire or rebuild the tree as a first step.6364## Canonical recovery paths6566| Intent | Path |67|---|---|68| Tracked stop (ownership-validated pid only) | `hipfire stop` |69| Stop + orphan reap + free port | `hipfire stop --force` (**approval** + owner inspection first); optional `--all` also reaps quantize jobs |70| Stop then start with same flags | `hipfire restart -d` (**approval** — force-reap semantics on the stop leg; inspect port/process owners first) |71| Scripted kill/free/optional relaunch | `scripts/serve-restart.sh [port] [--kill-only] [-- serve args…]` — destructive; see script header + [`docs/SERVE.md`](../../../docs/SERVE.md) |72| Read-only / diagnostic GPU/install inventory | `hipfire diag` or `.agents/skills/hipfire-diag/` (GPU/JIT side effects disclosed there) |73| Serve API / lifecycle prose | [`docs/SERVE.md`](../../../docs/SERVE.md), [`docs/CLI.md`](../../../docs/CLI.md) |74| Config / env owners | [`docs/CONFIG.md`](../../../docs/CONFIG.md), [`docs/env-vars.md`](../../../docs/env-vars.md) |75| Claim → validation route | [`docs/VALIDATION.md`](../../../docs/VALIDATION.md) |7677Default bind: `0.0.0.0:11435`. Pid/log: `~/.hipfire/serve.pid`,78`~/.hipfire/serve.log`. Install root: `~/.hipfire/` (`bin/daemon`, `models/`,79`config.toml`).8081## References in this skill8283| File | Role |84|---|---|85| `triage.sh` | Structured evidence gatherer (`--json` optional) |86| `playbook.md` | Symptom → diagnosis → repair catalog |87| `known-issues.md` | Dated current + historical/unknown caveats |88| `bisection.md` | Localize hangs/regressions after the catalog |8990## Out of scope9192- Performance claims or route admissions (methodology + `VALIDATION.md` /93 `docs/admissions.yml` only).94- Blind coherence-gate runs: `scripts/coherence-gate-*.sh` are **historical95 reproduction only**, not acceptance ([`docs/VALIDATION.md`](../../../docs/VALIDATION.md)).96- Editing product code unless the user asked for a code fix; this skill is97 runtime ops first.
warpfront