Storage Watchdog Ops

storage-watchdog-ops

Operator runbook for the ACFS storage watchdog — the Go daemon (acfs-storage-watchdog.service, built from ~/acfs/ops/storage-watchdog). This skill is the human/agent trigger and remediation layer over that daemon; it does not reimplement cleanup. When you need to know is the disk under pressure, did the watchdog handle it, and what do I do if it didn't — this is the runbook.

What the daemon actually does (so you interpret it correctly)

• It is a narrow, safe disk-reclaimer. It deletes only directories literally named target whose parent directory contains Cargo.toml (Rust build artifacts). Nothing else is ever removed.
• It does not follow symlinks, never deletes the filesystem root, ., or "", and never touches source files or non-Rust target/ directories.
• Cleanup starts only under pressure: free space below --min-free-gb (service default 50) OR used percent at/above --max-used-pct (default 90). With no pressure it logs ok: no pressure and deletes nothing.
• Under pressure it sorts candidates oldest-mtime first and deletes until free space reaches --target-free-gb (default 100) or candidates are exhausted, then logs cleanup complete.
• It runs as a user systemd service, Nice=19 / IOSchedulingClass=idle, ticking every --interval-seconds (default 600). Default scan roots: ~/dev ~/acfs ~/.cargo.

Thresholds are also overridable by env (STORAGE_WATCHDOG_MIN_FREE_GB, STORAGE_WATCHDOG_MAX_USED_PCT, STORAGE_WATCHDOG_TARGET_FREE_GB, STORAGE_WATCHDOG_INTERVAL_SECONDS, STORAGE_WATCHDOG_STATE_DIR, STORAGE_WATCHDOG_LOG_FILE).

1. Check status

# Is the daemon alive and ticking?
systemctl --user status acfs-storage-watchdog.service

# Recent decisions from the service journal
journalctl --user -u acfs-storage-watchdog.service -n 50 --no-pager

# Persistent decision log (survives restarts)
tail -n 50 "${STORAGE_WATCHDOG_LOG_FILE:-$HOME/.local/state/acfs-storage-watchdog/watchdog.log}"

# Ground truth on the disk it is protecting
df -h "$HOME/dev" "$HOME/acfs" 2>/dev/null

Key log lines to find (each tick emits a check):

storage-watchdog: check avail_kb=… used_pct=… min_free_gb=50 max_used_pct=90 target_free_gb=100 dry_run=false roots=…
storage-watchdog: ok: no pressure
storage-watchdog: delete target size_kb=… mtime=… path=…
storage-watchdog: cleanup complete candidates=… deleted_kb=… avail_kb=…

2. Interpret

The watchdog only ever frees Rust target/ space. If df says the disk is full of media, logs, container layers, or a single huge non-Rust dir, a healthy watchdog will correctly do nothing — that is not a watchdog bug, it is a different remediation (see system-performance-remediation).

3. Remediate

Restart a down daemon:

systemctl --user restart acfs-storage-watchdog.service
systemctl --user status acfs-storage-watchdog.service
journalctl --user -u acfs-storage-watchdog.service -n 20 --no-pager

Force an immediate, non-destructive assessment (does not wait for the next tick; deletes nothing — shows exactly what it would remove, oldest first):

cd "$HOME/acfs/ops/storage-watchdog"
go run ./cmd/storage-watchdog --once --dry-run --root "$HOME/dev" --root "$HOME/acfs"

Force a real one-shot cleanup (same safety policy; only Rust targets):

cd "$HOME/acfs/ops/storage-watchdog"
go run ./cmd/storage-watchdog --once --root "$HOME/dev" --root "$HOME/acfs"

Prove the safety policy still holds before trusting a forced run (creates a throwaway fixture, asserts old/new Rust targets deleted but source + non-Rust target/ preserved):

cd "$HOME/acfs/ops/storage-watchdog"
go run ./cmd/storage-watchdog --self-test

Hand-clean a single Rust target the daemon flagged but couldn't remove (verify it really is a Rust target first — never blind rm -rf):

p="<path from the delete-failed log line>"
test -f "$(dirname "$p")/Cargo.toml" && test "$(basename "$p")" = target \
  && du -sh "$p" && rm -rf "$p"   # only after both tests pass

Adjust thresholds (e.g. start cleaning earlier on a small disk) — edit the unit's ExecStart flags or set env, then reload:

systemctl --user edit acfs-storage-watchdog.service   # override min/max/target or roots
systemctl --user daemon-reload
systemctl --user restart acfs-storage-watchdog.service

4. Escalate

Escalate beyond this skill when:

• Pressure persists after a real --once run and cleanup complete shows candidates=0 or deleted_kb far below what's needed — the disk is full of something the watchdog is (correctly) not allowed to touch. Hand off to system-performance-remediation to find the actual hog (du -xh … | sort -h, container/image layers, logs, datasets).
• Repeated delete failed on the same path → a permissions/ownership or busy-mount issue the watchdog can't resolve; needs host admin.
• Daemon won't stay up (Restart=always but flapping) → inspect journalctl for the FATAL/exit reason; likely bad flags or absent roots.
• The host is a non-ACFS / non-Rust box where this daemon was mis-deployed — disable it (systemctl --user disable --now acfs-storage-watchdog.service) rather than fighting candidates=0.

Never widen the deletion policy as a "fix" (deleting non-target dirs, removing the Cargo.toml-parent check, or following symlinks). The narrow policy is the safety guarantee; broadening it turns a reclaimer into a data-loss incident.

Related

• system-performance-remediation — the broader disk/CPU/memory remediation rule this daemon was specialized from; use it when the watchdog has no candidates.
• ACFS source of truth: ~/acfs/ops/storage-watchdog/README.md and cmd/storage-watchdog/.