ramsbaby OpenClaw Self-Healing System
"The system that heals itself — or calls for help when it can't."
A five-level autonomous self-healing system for OpenClaw Gateway.
Architecture
Level 0: Preflight → Validate binary/.env/JSON, then exec the gateway
Level 1: KeepAlive → Instant restart (owned by the gateway service unit)
Level 2: Watchdog (180s) → PID + HTTP + memory, exponential backoff
Health Check (300s) → HTTP 200 + 3 retries → Level 3 escalation
Level 3: Claude Recovery → 30min AI-powered diagnosis in a tmux PTY session
Level 4: Multi-channel Alert → Human escalation
What's Special
- Claude Code CLI as the Level 3 emergency doctor, with a mandatory "read real state before diagnosing" prompt block to curb hallucinated fixes
- Preflight validation (Level 0) — catches config corruption before the gateway starts
- Persistent learning — recovery documentation (symptom → cause → solution → prevention)
- Reasoning logs — the AI's decision-making process is written to disk
- Multi-channel alerts — Discord, Slack and Telegram via
scripts/lib/notify.sh - Prometheus metrics — eight gauges for Grafana dashboards and alert rules
- Production-tested: 9 of 14 real incidents resolved autonomously (64%)
- macOS LaunchAgent and Linux systemd integration
Quick Setup
Recommended: one-line installer
curl -fsSL https://raw.githubusercontent.com/Ramsbaby/openclaw-self-healing/main/install.sh | bash
Preview first with bash -s -- --dry-run. The installer handles dependencies check,
directories, script download, .env generation and LaunchAgent/systemd registration.
Manual setup
1. Install dependencies
brew install tmux
npm install -g @anthropic-ai/claude-code
2. Configure environment
cp .env.example ~/.openclaw/.env
chmod 600 ~/.openclaw/.env
nano ~/.openclaw/.env # add your webhook and gateway token
3. Install scripts
Scripts must live where the LaunchAgents and escalation paths expect them:
DEST=~/.openclaw/skills/openclaw-self-healing/scripts
mkdir -p "$DEST/lib"
cp scripts/*.sh "$DEST/"
cp scripts/lib/*.sh "$DEST/lib/"
chmod 700 "$DEST"/*.sh "$DEST"/lib/*.sh
cp launchagent/com.openclaw.healthcheck.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
4. Verify
launchctl list | grep openclaw.healthcheck
bash scripts/validate-deployment.sh
tail -f ~/.openclaw/logs/watchdog.log
Scripts
| Script | Level | Description |
|---|---|---|
gateway-preflight.sh |
0 | Config validation before start; execs the gateway on success |
gateway-watchdog.sh |
2 | PID + HTTP + memory monitoring, backoff, doctor --fix, L3 escalation |
gateway-healthcheck.sh |
2 | HTTP 200 check + 3 retries + escalation |
emergency-recovery-v2.sh |
3 | Claude PTY session with learning + reasoning logs (the one L0/L2 call) |
emergency-recovery.sh |
3 | Earlier v1 recovery script; not installed by the installers |
emergency-recovery-monitor.sh |
4 | Alerts when a recovery session fails |
lib/notify.sh |
4 | Discord / Slack / Telegram dispatcher |
lib/llm-gateway.sh |
— | LLM provider router; not yet called by any script |
validate-deployment.sh |
ops | Post-install verification of all five levels |
incident-digest.sh |
ops | Weekly incident report |
prometheus-exporter.py |
ops | Metrics HTTP server (requires python3) |
Configuration
All settings via environment variables in ~/.openclaw/.env:
| Variable | Default | Description |
|---|---|---|
DISCORD_WEBHOOK_URL |
(none) | Discord webhook for alerts |
SLACK_WEBHOOK_URL |
(none) | Slack webhook, lib/notify.sh only |
NOTIFICATION_CHANNEL |
auto-detect | Force discord / slack / telegram |
OPENCLAW_GATEWAY_URL |
http://localhost:18789/ |
Gateway health check URL |
OPENCLAW_GATEWAY_TOKEN |
(none) | Required by Level 0 preflight |
ANTHROPIC_API_KEY |
(none) | Required by Levels 0 and 3 |
HEALTH_CHECK_MAX_RETRIES |
3 |
Restart attempts before escalation |
OPENCLAW_WATCHDOG_ESCALATE_TO_L3_AFTER |
1800 |
Continuous-failure seconds before Level 3 |
EMERGENCY_RECOVERY_TIMEOUT |
1800 |
Claude recovery timeout (30 min) |
See .env.example for the full annotated list.
Testing
Test Level 2 (Health Check)
# Run manually
bash ~/.openclaw/skills/openclaw-self-healing/scripts/gateway-healthcheck.sh
# Expected output:
# ✅ Gateway healthy
Test Level 3 (Claude Recovery)
# Inject a config error (backup first!)
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak
# Wait for Health Check to detect and escalate (~8 min)
tail -f ~/openclaw/memory/emergency-recovery-*.log
Links
- GitHub: https://github.com/Ramsbaby/openclaw-self-healing
- Docs: https://github.com/Ramsbaby/openclaw-self-healing/tree/main/docs
License
MIT License - do whatever you want with it.
Built by @ramsbaby + Jarvis 🦞