Incident Responder
Standardizes failure handling. No more ad-hoc.
Classes
| Class | Trigger | First action |
|---|---|---|
cron-failure |
cron_health.py reports >26h gap |
Run cron manually, capture output |
cron-cascade |
detect_cascade() returns true |
Suspect shared config; check Vertex proxy first |
gateway-499 |
WA gateway 499 disconnect >10min | docker restart ocana-litellm-proxy |
semantic-stale |
check_sqlite_freshness() False |
Re-run memory_search ingest pipeline |
pa-failure |
PA agent error reported | DM owner; do NOT message PA's owner directly |
Procedure (every class)
- Classify using table above
- Diagnose — collect logs (
tools/error_tracker.py log <component> <msg>) - Notify — DM the owner ONLY (not the affected PA's owner)
- Mitigate — execute the first action from the table
- Log — append daily-note:
[HH:MM IL] INCIDENT <class>: <one-line summary>; resolved=<yes/no> - Postmortem (if user-visible impact >5min) — create
memory/incidents/YYYY-MM-DD-<class>.md
Anti-patterns
- Do NOT loop trying the same fix twice
- Do NOT tell the PA's owner there's a problem before the owner knows
- Do NOT skip the daily-note log entry