OpenEvidence Incident Runbook
Table of Contents
Overview
Rapid incident response procedures for OpenEvidence clinical AI integration outages in healthcare environments. Includes severity classification, triage steps, error-specific remediation, fallback procedures, and postmortem templates.
Prerequisites
- Access to OpenEvidence status page
- kubectl access to production cluster
- Prometheus/Grafana access
- PagerDuty or on-call system access
Severity Levels
| Level |
Definition |
Response Time |
Examples |
| P1 |
Complete outage |
< 15 min |
API unreachable, all queries failing |
| P2 |
Degraded service |
< 1 hour |
High latency, partial failures |
| P3 |
Minor impact |
< 4 hours |
DeepConsult delays, webhook issues |
| P4 |
No user impact |
Next business day |
Alert noise, logging issues |
Critical Note
OpenEvidence outages may affect clinical decision-making. Always communicate clearly with clinical staff, ensure fallback procedures are known, and document any clinical impact.
Instructions
Step 1: Initial Assessment (2 minutes)
Check OpenEvidence status page, your integration health endpoint, error rate metrics, and recent error logs.
Step 2: Follow Decision Tree
- API errors + OpenEvidence status incident -> Enable fallback, wait for resolution
- API errors + no status incident -> Check credentials, config, network
- No API errors + unhealthy service -> Infrastructure issue (pods, memory, network)
Step 3: Apply Error-Specific Remediation
- 401/403: Verify API key, rotate if needed, restart pods
- 429: Enable request queuing, contact OpenEvidence for limit increase
- 500/503: Enable graceful degradation, notify clinical staff
- Timeout: Increase timeout temporarily, check network latency
Step 4: Enable Fallback
Return helpful message directing to UpToDate, DynaMed, or clinical guidelines directly.
Step 5: Communicate
Notify clinical staff via Slack/Teams and email. Update status page.
Step 6: Post-Incident
Collect evidence (logs, metrics, alerts), run postmortem with clinical impact assessment, create action items.
Output
- Quick triage procedure completed
- Issue identified and categorized
- Remediation applied
- Clinical staff notified
- Evidence collected for postmortem
Error Handling
| Error Type |
Quick Fix |
| 401/403 Auth |
kubectl create secret with new key, kubectl rollout restart |
| 429 Rate Limit |
kubectl set env RATE_LIMIT_MODE=queue |
| 500/503 Server |
kubectl set env OPENEVIDENCE_FALLBACK=true |
| Timeout |
kubectl set env OPENEVIDENCE_TIMEOUT=60000 |
Examples
One-Line Health Check
set -euo pipefail
curl -sf https://api.yourhealthcare.com/health/openevidence | jq '.status' || echo "UNHEALTHY"
Enable/Disable Fallback
set -euo pipefail
kubectl set env deployment/clinical-evidence-api OPENEVIDENCE_FALLBACK=true # Enable
kubectl set env deployment/clinical-evidence-api OPENEVIDENCE_FALLBACK=false # Disable
See detailed implementation for advanced patterns.
Resources
1---2name: openevidence-incident-runbook3description: Execute OpenEvidence incident response procedures with triage, mitigation, and postmortem. Use when responding to OpenEvidence-related outages, investigating errors, or running post-incident reviews for clinical AI integration failures. Trigger with phrases like "openevidence incident", "openevidence outage", "openevidence down", "openevidence emergency", "clinical ai broken".4license: MIT5---6# OpenEvidence Incident Runbook78## Table of Contents9- [Overview](#overview)10- [Prerequisites](#prerequisites)11- [Instructions](#instructions)12- [Output](#output)13- [Error Handling](#error-handling)14- [Examples](#examples)15- [Resources](#resources)1617## Overview18Rapid incident response procedures for OpenEvidence clinical AI integration outages in healthcare environments. Includes severity classification, triage steps, error-specific remediation, fallback procedures, and postmortem templates.1920## Prerequisites21- Access to OpenEvidence status page22- kubectl access to production cluster23- Prometheus/Grafana access24- PagerDuty or on-call system access2526## Severity Levels2728| Level | Definition | Response Time | Examples |29|-------|------------|---------------|----------|30| P1 | Complete outage | < 15 min | API unreachable, all queries failing |31| P2 | Degraded service | < 1 hour | High latency, partial failures |32| P3 | Minor impact | < 4 hours | DeepConsult delays, webhook issues |33| P4 | No user impact | Next business day | Alert noise, logging issues |3435## Critical Note36OpenEvidence outages may affect clinical decision-making. Always communicate clearly with clinical staff, ensure fallback procedures are known, and document any clinical impact.3738## Instructions3940### Step 1: Initial Assessment (2 minutes)41Check OpenEvidence status page, your integration health endpoint, error rate metrics, and recent error logs.4243### Step 2: Follow Decision Tree44- API errors + OpenEvidence status incident -> Enable fallback, wait for resolution45- API errors + no status incident -> Check credentials, config, network46- No API errors + unhealthy service -> Infrastructure issue (pods, memory, network)4748### Step 3: Apply Error-Specific Remediation49- **401/403**: Verify API key, rotate if needed, restart pods50- **429**: Enable request queuing, contact OpenEvidence for limit increase51- **500/503**: Enable graceful degradation, notify clinical staff52- **Timeout**: Increase timeout temporarily, check network latency5354### Step 4: Enable Fallback55Return helpful message directing to UpToDate, DynaMed, or clinical guidelines directly.5657### Step 5: Communicate58Notify clinical staff via Slack/Teams and email. Update status page.5960### Step 6: Post-Incident61Collect evidence (logs, metrics, alerts), run postmortem with clinical impact assessment, create action items.6263## Output64- Quick triage procedure completed65- Issue identified and categorized66- Remediation applied67- Clinical staff notified68- Evidence collected for postmortem6970## Error Handling71| Error Type | Quick Fix |72|------------|-----------|73| 401/403 Auth | `kubectl create secret` with new key, `kubectl rollout restart` |74| 429 Rate Limit | `kubectl set env RATE_LIMIT_MODE=queue` |75| 500/503 Server | `kubectl set env OPENEVIDENCE_FALLBACK=true` |76| Timeout | `kubectl set env OPENEVIDENCE_TIMEOUT=60000` |7778## Examples7980### One-Line Health Check81```bash82set -euo pipefail83curl -sf https://api.yourhealthcare.com/health/openevidence | jq '.status' || echo "UNHEALTHY"84```8586### Enable/Disable Fallback87```bash88set -euo pipefail89kubectl set env deployment/clinical-evidence-api OPENEVIDENCE_FALLBACK=true # Enable90kubectl set env deployment/clinical-evidence-api OPENEVIDENCE_FALLBACK=false # Disable91```9293See [detailed implementation](${CLAUDE_SKILL_DIR}/references/implementation.md) for advanced patterns.9495## Resources96- [OpenEvidence Status](https://status.openevidence.com/)97- [OpenEvidence Support](mailto:support@openevidence.com)
micsapp