stevengonsalvez ainb fleet:daemon
DEPRECATED. Use
/ainb-fleet:atcinlitemode.ainb fleet atc mode <name> --set litegives you this exact loop — the same 5-second scan, the same auto-continueon the same known transient errors — owned by the ATC supervisor, inside its per-session retry cap and its safety ledger, and structurally unable to run beside the LLM heartbeat.This standalone daemon REFUSES to start while ATC owns the fleet in EITHER mode, because both send the same auto-
continueto the same pane and each de-dups only inside its own process, so the pane gets it twice. Pass--force-raceto run both anyway.The difference that matters is the retry cap: ATC gives up on a session after a bounded number of attempts and escalates to you. This daemon has no cap, so a session that keeps failing is retried forever, and running it beside ATC makes ATC's cap meaningless.
Kept for unmanaged one-off recovery, not removed.
Superseded by
/ainb-fleet:atcfor managed fleets. ATC absorbs this daemon's job — its ERR playbook does the same auto-continue, but with a per-session retry cap + escalate-on-exhaustion that this daemon lacks (see the sharp edge below). Preferainb fleet atc setup <name>for unattended supervision. The daemon stays for unmanaged one-off recovery; do not run it against sessions an ATC instance already manages — they race.
Background watcher. Reads each session's tmux pane (capture-pane) and
auto-continues sessions hitting API errors via tmux send-keys. It does
not register as a peer — it is purely tmux-driven (broker health is
checked only to print an informational line).
Run
ainb fleet daemon # quiet
ainb fleet daemon --verbose # log every detection + send
For real background use:
nohup ainb fleet daemon --verbose > ~/.ainb-fleet.log 2>&1 &
What it does each tick (every 5s)
- Re-discover all sessions (ainb + peers + jobs, merged + deduped).
- For each tmux-bearing session:
tmux capture-pane -p -S -80(last 80 lines). - Run the API-error regex set over the buffer.
- If a match fires AND the (session_id, pattern, snippet-tail) dedupe
key hasn't been seen yet → send
continueto that session via the standard route (tmux-first by default — seeAINB_FLEET_TRANSPORT).
Detected error patterns
| name | regex |
|---|---|
rate_limited |
\brate[_ ]limited\b |
overloaded |
\boverloaded_error\b | \bModel is overloaded\b |
internal_server_error |
\binternal_server_error\b |
request_timeout |
\brequest timed out\b |
socket_hang_up |
\bsocket hang up\b |
fetch_failed |
\bAPI Error\b | \bfetch failed\b |
connection_reset |
\bECONNRESET\b | \bconnection reset\b |
Case-insensitive. Word-boundaried to avoid false positives.
Dedupe key
key = (session_id, pattern_name, last_40_chars_of_match_context)
Within a single daemon run, the same key only fires continue once.
This prevents spam when the error text scrolls up but is still in the
buffer.
⚠️ Sharp edge — no retry cap in v0.1
If a session is permanently broken (wrong credentials, model deprecated,
loop bug), the daemon will keep firing continue forever as new errors
match new dedupe keys. Watch the daemon log; kill it (Ctrl-C or
kill <pid>) when you see runaway repetition.
Roadmap: per-session retry cap with exponential backoff.
Stop the daemon
# foreground: Ctrl-C
# background:
pkill -INT -f "ainb fleet daemon"
# or kill by exact PID if you know it
kill <pid>
The daemon holds no broker registration, so there is no stale peer entry to
clean up on exit — Ctrl-C / kill is enough.
Observe what the daemon is doing
On start the daemon prints whether the broker is reachable (informational only — writes go via tmux regardless):
[fleet/daemon] broker healthy at 127.0.0.1:7899
# or, when the broker is down:
[fleet/daemon] broker not reachable — operating in tmux-only mode
With --verbose, every detection also prints:
[fleet/daemon] auto-continue -> <tmux_session> (<pattern>)
Tail the nohup log to watch in real time:
tail -F ~/.ainb-fleet.log
When NOT to run the daemon
- During hand-debugging — the daemon will send
continuewhile you're reading the error, racing your investigation. - On a fleet doing batch work that explicitly throws errors as control flow — the daemon will misinterpret intentional errors.
- On a session with no live tmux pane (bg job, dead tmux) — there is
nothing to send-keys to; under the default
tmux-firstit can only reach such a session if it still has a healthy broker peer.