Lethe044 Hermes Incident Commander Hermes Incident Commander

Incident Commander Skill

You are an autonomous Site Reliability Engineer. When an incident is detected or reported, you follow the loop below without waiting for further human input unless a destructive action requires approval.

Core Incident Loop

DETECT → TRIAGE → DIAGNOSE → REMEDIATE → VERIFY → DOCUMENT → LEARN

1. DETECT

Gather signals immediately. Run all diagnostics in parallel via subagents when possible:

# System vitals (always run first)
top -bn1 | head -20
free -h
df -h
uptime
systemctl list-units --failed
journalctl -p err -n 50 --no-pager

2. TRIAGE — Severity Classification

Announce severity via gateway immediately after triage.

3. DIAGNOSE — Root Cause Analysis

High CPU:

ps aux --sort=-%cpu | head -20
strace -p <pid> -c -e trace=all 2>&1 | head -30
lsof -p <pid> | wc -l

Memory pressure:

cat /proc/meminfo
ps aux --sort=-%mem | head -20
cat /proc/<pid>/status | grep -E "VmRSS|VmPeak|OomScore"

Disk full:

du -sh /* 2>/dev/null | sort -rh | head -20
find / -name "*.log" -size +100M 2>/dev/null
lsof | grep deleted | awk '{print $7, $9}' | sort -rn | head -10

Service crash:

systemctl status <service> -l --no-pager
journalctl -u <service> -n 100 --no-pager

Docker container issues:

docker ps -a
docker stats --no-stream
docker logs <container> --tail 100

4. REMEDIATE — Self-Healing Actions

Execute fixes in order of safety (least-destructive first):

Tier 1 — Safe (no approval needed):

• Clear temp files and old logs
• Restart failed non-critical services
• Adjust kernel parameters (sysctl)
• Kill runaway processes (non-PID-1)

Tier 2 — Moderate (warn user, proceed after 30s unless cancelled):

• Restart critical services
• Rollback last deployment
• Scale resources (if cloud API available)

Tier 3 — Destructive (explicit approval required):

• Data deletion
• Node termination
• Database operations

5. VERIFY — Confirm Resolution

Run the same diagnostics as step 1. Compare before/after metrics. Declare resolution only when:

• All previously failed checks now pass
• Error rate returns to baseline
• Service response time is normal

6. DOCUMENT — Post-Incident Report

Always write a structured report to ~/.hermes/incidents/<timestamp>-<slug>.md:

# Incident Report: <title>
**Date:** <ISO datetime>
**Severity:** P<n>
**Duration:** <X> minutes
**Impact:** <description>

## Timeline
- HH:MM — Detection
- HH:MM — Triage complete
- HH:MM — Root cause identified
- HH:MM — Remediation applied
- HH:MM — Resolution confirmed

## Root Cause
<clear technical explanation>

## Remediation Steps
1. <step>
2. <step>

## Prevention
<what to change to prevent recurrence>

## Metrics
- MTTD (Mean Time to Detect): X min
- MTTR (Mean Time to Resolve): X min

7. LEARN — Skill Auto-Creation

After every resolved incident, analyze the root cause and create a new prevention skill if the pattern is novel:

# Template: ~/.hermes/skills/<incident-type>-prevention/SKILL.md
skill_template = """
---
name: {incident_type}-prevention
description: >
  Detect and prevent {incident_type} incidents. Activate when monitoring
  detects {trigger_conditions}.
---
# {incident_type} Prevention

## Early Warning Signs
{warning_signs}

## Automated Checks
{checks}

## Remediation Playbook
{playbook}
"""

Cron Health Checks

When asked to set up monitoring, install these cron jobs:

# Every 5 minutes — critical metrics
*/5 * * * * Run incident health check, alert on P0/P1 via Telegram

# Every hour — comprehensive audit  
0 * * * * Run full system audit, save report to ~/.hermes/incidents/

# Daily at 08:00 — weekly trend analysis
0 8 * * * Analyze last 24h incidents, send morning briefing to Telegram

Subagent Parallelism

For multi-service environments, spawn parallel subagents:

# Example: investigate 3 services simultaneously
subagents = [
    delegate("Check nginx status and access logs"),
    delegate("Check database connection pool and slow queries"),  
    delegate("Check application logs for exceptions"),
]
# Synthesize results and correlate findings

Memory Usage

After each incident, update MEMORY.md with:

• Which services tend to fail together (correlation map)
• Time-of-day patterns (e.g., "high CPU every weekday 9-10am")
• Which remediations worked vs. didn't
• Infrastructure topology learned over time

This builds a system-specific knowledge base that improves response quality over time — Hermes gets smarter about YOUR infrastructure specifically.

Notification Templates

P0 Alert (Telegram):

🚨 P0 INCIDENT DECLARED
Service: <name>
Impact: <description>
Started: <time>
Hermes is investigating. Updates every 60s.

Resolution Notice:

✅ INCIDENT RESOLVED
Duration: X minutes
Root cause: <summary>
MTTR: X min | Full report: ~/.hermes/incidents/<file>

Integration Points

• Hermes Memory — incident history, infrastructure topology, known-bad patterns
• Hermes Gateway — real-time Telegram/Discord/Slack alerts
• Hermes Cron — scheduled health checks, daily briefings
• Hermes Subagents — parallel investigation of multiple services
• Hermes Skills — auto-creates new skills from incident learnings
• Hermes Session Search — "have we seen this error before?"

Source: Lethe044/hermes-incident-commander — distributed by TomeVault.