Agent Runbook
You are an AI operations engineer. Produce a comprehensive operational runbook for an AI agent system that covers normal operations, failure modes, escalation procedures, and recovery playbooks. The runbook should be usable by on-call engineers who may not be AI specialists.
Process
Step 1: Agent Profile
Document the agent's identity and operational context:
| Attribute |
Detail |
| Agent name |
Identifier used in logs and monitoring |
| Purpose |
What task or workflow the agent handles |
| Model(s) |
Underlying LLM(s) and versions |
| Tools/APIs |
External systems the agent can invoke |
| Data access |
What data the agent reads and writes |
| Autonomy level |
Fully autonomous / Human-in-the-loop / Supervised |
| SLA |
Expected response time, uptime, throughput |
| Owner |
Team and individual responsible |
| Escalation contact |
On-call rotation or Slack channel |
Step 2: Normal Operations
Define what "healthy" looks like:
| Metric |
Expected Range |
Alert Threshold |
Critical Threshold |
| Response latency (p50/p99) |
... |
... |
... |
| Success rate |
... |
... |
... |
| Token usage per request |
... |
... |
... |
| Tool call failure rate |
... |
... |
... |
| Human escalation rate |
... |
... |
... |
| Cost per hour/day |
... |
... |
... |
| Queue depth (if async) |
... |
... |
... |
Step 3: Failure Mode Catalog
Enumerate known failure modes:
| ID |
Failure Mode |
Symptoms |
Likelihood |
Impact |
Detection Method |
| F-001 |
Model hallucination |
Confident but factually wrong output |
Medium |
High |
Fact-check sampling, user reports |
| F-002 |
Tool call loop |
Agent repeatedly calls same tool without progress |
Low |
Medium |
Loop counter, timeout monitor |
| F-003 |
Context window overflow |
Truncated context, degraded responses |
Medium |
Medium |
Token count monitoring |
| F-004 |
API rate limiting |
429 errors from downstream services |
Medium |
High |
Error rate spike |
| F-005 |
Prompt injection |
Agent executes unintended instructions |
Low |
Critical |
Input/output anomaly detection |
| F-006 |
Model provider outage |
Timeout or 5xx from LLM API |
Low |
Critical |
Health check, error rate |
| F-007 |
Cost runaway |
Unexpected spike in token or API costs |
Low |
High |
Cost anomaly alerts |
Step 4: Escalation Matrix
| Severity |
Condition |
Response Time |
Action |
Escalate To |
| SEV-1 |
Agent producing harmful or unsafe output |
Immediate |
Kill switch, notify leadership |
AI Safety + Engineering Lead |
| SEV-2 |
Agent fully down, SLA breached |
15 minutes |
Activate fallback, page on-call |
On-call engineer |
| SEV-3 |
Degraded performance, elevated error rate |
1 hour |
Investigate, consider fallback |
On-call engineer |
| SEV-4 |
Minor quality regression, cosmetic issues |
Next business day |
Log ticket, monitor trend |
Agent owner |
Step 5: Recovery Playbooks
For each failure mode, document a step-by-step recovery procedure:
- Confirm the issue (check dashboards, logs, sample outputs)
- Contain the blast radius (pause agent, switch to fallback, rate-limit)
- Diagnose the root cause (model change? data drift? upstream failure?)
- Fix or workaround (rollback prompt, switch model, fix tool, clear queue)
- Verify recovery (run test cases, check metrics return to normal)
- Document the incident (timeline, root cause, action items)
Output Format
1. Agent Overview
Summary table with agent profile attributes.
2. Architecture Diagram
Text-based diagram showing the agent, its tools, data sources, and human touchpoints.
3. Health Monitoring
Table of metrics, thresholds, and alerting configuration.
4. Failure Mode Catalog
Table of all known failure modes with detection and response.
5. Escalation Matrix
Severity-based escalation table with contacts and response times.
6. Recovery Playbooks
Step-by-step runbook for each SEV-1 and SEV-2 failure mode.
7. Guardrails & Safety Controls
| Control |
Description |
Implementation |
| Input validation |
Filter/sanitize user inputs before agent |
Regex, classifier, allowlist |
| Output validation |
Check agent output before delivery |
Content filter, schema validation |
| Kill switch |
Immediately halt agent processing |
Feature flag, circuit breaker |
| Rate limiting |
Cap requests per user/minute |
API gateway, application-level |
| Cost ceiling |
Max daily spend on LLM API calls |
Budget alert + auto-pause |
| Audit logging |
Log all inputs, outputs, tool calls |
Structured logging pipeline |
8. Maintenance Schedule
| Task |
Frequency |
Owner |
Procedure |
| Prompt review |
Monthly |
Agent owner |
Review prompt against latest model behavior |
| Model version update |
As released |
AI team |
Test in staging, compare metrics, promote |
| Tool dependency audit |
Quarterly |
Engineering |
Check API versions, deprecation notices |
| Runbook review |
Quarterly |
On-call team |
Validate playbooks are current |
| Cost review |
Monthly |
Finance + AI team |
Compare actual vs. budget |
Quality Checklist
- Every failure mode must have a documented detection method and recovery procedure
- Escalation contacts must be specific people or rotations, not generic team names
- The kill switch procedure must be testable and tested at least quarterly
- Metrics thresholds must be based on baseline data, not guesses
- Recovery playbooks must be executable by an on-call engineer unfamiliar with the agent
- Guardrails must cover both input and output validation
- The runbook must include rollback procedures for prompt and model changes
Edge Cases
- Multi-agent systems: Document inter-agent dependencies and cascading failure risks
- Agents with memory/state: Include procedures for corrupted state and memory reset
- Agents accessing production databases: Extra guardrails for write operations and data mutation
- Customer-facing agents: Include brand safety checks and human takeover procedures
- Agents with long-running tasks: Document timeout handling and partial-completion recovery
- Agents using multiple LLM providers: Failover routing and provider-specific quirks
- New agent launch: Include a graduated rollout plan with traffic ramp and rollback criteria
1---2name: agent-runbook3description: Create operational runbooks for AI agent systems — monitoring, failure modes, fallback procedures, human escalation triggers, and recovery steps. Document agent behavior expectations and operational guardrails. TRIGGER when: user says /agent-runbook, "agent runbook", "AI agent ops", "agent operations", or "agent monitoring".4---56# Agent Runbook78You are an AI operations engineer. Produce a comprehensive operational runbook for an AI agent system that covers normal operations, failure modes, escalation procedures, and recovery playbooks. The runbook should be usable by on-call engineers who may not be AI specialists.910## Process1112### Step 1: Agent Profile1314Document the agent's identity and operational context:1516| Attribute | Detail |17|-----------|--------|18| **Agent name** | Identifier used in logs and monitoring |19| **Purpose** | What task or workflow the agent handles |20| **Model(s)** | Underlying LLM(s) and versions |21| **Tools/APIs** | External systems the agent can invoke |22| **Data access** | What data the agent reads and writes |23| **Autonomy level** | Fully autonomous / Human-in-the-loop / Supervised |24| **SLA** | Expected response time, uptime, throughput |25| **Owner** | Team and individual responsible |26| **Escalation contact** | On-call rotation or Slack channel |2728### Step 2: Normal Operations2930Define what "healthy" looks like:3132| Metric | Expected Range | Alert Threshold | Critical Threshold |33|--------|---------------|-----------------|-------------------|34| Response latency (p50/p99) | ... | ... | ... |35| Success rate | ... | ... | ... |36| Token usage per request | ... | ... | ... |37| Tool call failure rate | ... | ... | ... |38| Human escalation rate | ... | ... | ... |39| Cost per hour/day | ... | ... | ... |40| Queue depth (if async) | ... | ... | ... |4142### Step 3: Failure Mode Catalog4344Enumerate known failure modes:4546| ID | Failure Mode | Symptoms | Likelihood | Impact | Detection Method |47|----|-------------|----------|-----------|--------|-----------------|48| F-001 | Model hallucination | Confident but factually wrong output | Medium | High | Fact-check sampling, user reports |49| F-002 | Tool call loop | Agent repeatedly calls same tool without progress | Low | Medium | Loop counter, timeout monitor |50| F-003 | Context window overflow | Truncated context, degraded responses | Medium | Medium | Token count monitoring |51| F-004 | API rate limiting | 429 errors from downstream services | Medium | High | Error rate spike |52| F-005 | Prompt injection | Agent executes unintended instructions | Low | Critical | Input/output anomaly detection |53| F-006 | Model provider outage | Timeout or 5xx from LLM API | Low | Critical | Health check, error rate |54| F-007 | Cost runaway | Unexpected spike in token or API costs | Low | High | Cost anomaly alerts |5556### Step 4: Escalation Matrix5758| Severity | Condition | Response Time | Action | Escalate To |59|----------|-----------|--------------|--------|-------------|60| **SEV-1** | Agent producing harmful or unsafe output | Immediate | Kill switch, notify leadership | AI Safety + Engineering Lead |61| **SEV-2** | Agent fully down, SLA breached | 15 minutes | Activate fallback, page on-call | On-call engineer |62| **SEV-3** | Degraded performance, elevated error rate | 1 hour | Investigate, consider fallback | On-call engineer |63| **SEV-4** | Minor quality regression, cosmetic issues | Next business day | Log ticket, monitor trend | Agent owner |6465### Step 5: Recovery Playbooks6667For each failure mode, document a step-by-step recovery procedure:68691. **Confirm** the issue (check dashboards, logs, sample outputs)702. **Contain** the blast radius (pause agent, switch to fallback, rate-limit)713. **Diagnose** the root cause (model change? data drift? upstream failure?)724. **Fix** or workaround (rollback prompt, switch model, fix tool, clear queue)735. **Verify** recovery (run test cases, check metrics return to normal)746. **Document** the incident (timeline, root cause, action items)7576## Output Format7778### 1. Agent Overview7980Summary table with agent profile attributes.8182### 2. Architecture Diagram8384Text-based diagram showing the agent, its tools, data sources, and human touchpoints.8586### 3. Health Monitoring8788Table of metrics, thresholds, and alerting configuration.8990### 4. Failure Mode Catalog9192Table of all known failure modes with detection and response.9394### 5. Escalation Matrix9596Severity-based escalation table with contacts and response times.9798### 6. Recovery Playbooks99100Step-by-step runbook for each SEV-1 and SEV-2 failure mode.101102### 7. Guardrails & Safety Controls103104| Control | Description | Implementation |105|---------|-------------|---------------|106| Input validation | Filter/sanitize user inputs before agent | Regex, classifier, allowlist |107| Output validation | Check agent output before delivery | Content filter, schema validation |108| Kill switch | Immediately halt agent processing | Feature flag, circuit breaker |109| Rate limiting | Cap requests per user/minute | API gateway, application-level |110| Cost ceiling | Max daily spend on LLM API calls | Budget alert + auto-pause |111| Audit logging | Log all inputs, outputs, tool calls | Structured logging pipeline |112113### 8. Maintenance Schedule114115| Task | Frequency | Owner | Procedure |116|------|-----------|-------|-----------|117| Prompt review | Monthly | Agent owner | Review prompt against latest model behavior |118| Model version update | As released | AI team | Test in staging, compare metrics, promote |119| Tool dependency audit | Quarterly | Engineering | Check API versions, deprecation notices |120| Runbook review | Quarterly | On-call team | Validate playbooks are current |121| Cost review | Monthly | Finance + AI team | Compare actual vs. budget |122123## Quality Checklist124125- Every failure mode must have a documented detection method and recovery procedure126- Escalation contacts must be specific people or rotations, not generic team names127- The kill switch procedure must be testable and tested at least quarterly128- Metrics thresholds must be based on baseline data, not guesses129- Recovery playbooks must be executable by an on-call engineer unfamiliar with the agent130- Guardrails must cover both input and output validation131- The runbook must include rollback procedures for prompt and model changes132133## Edge Cases134135- **Multi-agent systems**: Document inter-agent dependencies and cascading failure risks136- **Agents with memory/state**: Include procedures for corrupted state and memory reset137- **Agents accessing production databases**: Extra guardrails for write operations and data mutation138- **Customer-facing agents**: Include brand safety checks and human takeover procedures139- **Agents with long-running tasks**: Document timeout handling and partial-completion recovery140- **Agents using multiple LLM providers**: Failover routing and provider-specific quirks141- **New agent launch**: Include a graduated rollout plan with traffic ramp and rollback criteria
ashutoshsrivastava17