# Juicebox Incident Runbook

> Execute Juicebox incident response procedures. Use when responding to production incidents, troubleshooting outages, or following incident management protocols. Trigger with phrases like "juicebox incident", "juicebox outage", "juicebox down", "juicebox emergency".

- Skill: `tools-only/juicebox-incident-runbook` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds add tools-only/juicebox-incident-runbook`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/juicebox-incident-runbook/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/tools-only/juicebox-incident-runbook

---


# Juicebox Incident Runbook

## Overview
Standardized incident response procedures for Juicebox integration issues.

## Incident Severity Levels

| Severity | Description | Response Time | Examples |
|----------|-------------|---------------|----------|
| P1 | Critical | < 15 min | Complete outage, data loss |
| P2 | High | < 1 hour | Major feature broken, degraded performance |
| P3 | Medium | < 4 hours | Minor feature issue, workaround exists |
| P4 | Low | < 24 hours | Cosmetic, non-blocking |

## Quick Diagnostics

### Step 1: Immediate Assessment
```bash
#!/bin/bash
# quick-diag.sh - Run immediately when incident detected

echo "=== Juicebox Quick Diagnostics ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Check Juicebox status page
echo ""
echo "=== Juicebox Status ==="
curl -s https://status.juicebox.ai/api/status | jq '.status'

# Check our API health
echo ""
echo "=== Our API Health ==="
curl -s http://localhost:8080/health/ready | jq '.'

# Check recent error logs
echo ""
echo "=== Recent Errors (last 5 min) ==="
kubectl logs -l app=juicebox-integration --since=5m | grep -i error | tail -20

# Check metrics
echo ""
echo "=== Error Rate ==="
curl -s http://localhost:9090/api/v1/query?query=rate\(juicebox_requests_total\{status=\"error\"\}\[5m\]\) | jq '.data.result[0].value[1]'
```

### Step 2: Identify Root Cause
```markdown
## Incident Triage Decision Tree

1. Is Juicebox status page showing issues?
   - YES → External outage, skip to "External Outage Response"
   - NO → Continue

2. Are we getting authentication errors (401)?
   - YES → Check API key validity, skip to "Auth Issues"
   - NO → Continue

3. Are we getting rate limited (429)?
   - YES → Skip to "Rate Limit Response"
   - NO → Continue

4. Are requests timing out?
   - YES → Skip to "Timeout Response"
   - NO → Continue

5. Are we getting unexpected errors?
   - YES → Skip to "Application Error Response"
   - NO → Gather more data
```

## Response Procedures

### External Outage Response
```markdown
## When Juicebox is Down

1. **Confirm Outage**
   - Check https://status.juicebox.ai
   - Verify with curl test to API

2. **Enable Fallback Mode**
   ```bash
   kubectl set env deployment/juicebox-integration JUICEBOX_FALLBACK=true
   ```

3. **Notify Stakeholders**
   - Post to #incidents channel
   - Update status page if customer-facing

4. **Monitor Recovery**
   - Set up alert for Juicebox status change
   - Prepare to disable fallback mode

5. **Post-Incident**
   - Disable fallback when Juicebox recovers
   - Document timeline and impact
```

### Auth Issues Response
```markdown
## When Authentication Fails

1. **Verify API Key**
   ```bash
   # Mask key for logging
   echo "Key prefix: ${JUICEBOX_API_KEY:0:10}..."

   # Test auth
   curl -H "Authorization: Bearer $JUICEBOX_API_KEY" \
     https://api.juicebox.ai/v1/auth/me
   ```

2. **Check Key Status in Dashboard**
   - Log into https://app.juicebox.ai
   - Verify key is active and not revoked

3. **Rotate Key if Compromised**
   - Generate new key in dashboard
   - Update secret manager
   - Restart pods
   ```bash
   kubectl rollout restart deployment/juicebox-integration
   ```

4. **Verify Recovery**
   - Check health endpoint
   - Monitor error rate
```

### Rate Limit Response
```markdown
## When Rate Limited

1. **Check Current Usage**
   ```bash
   curl -H "Authorization: Bearer $JUICEBOX_API_KEY" \
     https://api.juicebox.ai/v1/usage
   ```

2. **Immediate Mitigation**
   - Enable aggressive caching
   - Reduce request rate
   ```bash
   kubectl set env deployment/juicebox-integration JUICEBOX_RATE_LIMIT=10
   ```

3. **If Quota Exhausted**
   - Contact Juicebox support for temporary increase
   - Implement request queuing

4. **Long-term Fix**
   - Review usage patterns
   - Implement better caching
   - Consider plan upgrade
```

### Timeout Response
```markdown
## When Requests Timeout

1. **Check Network**
   ```bash
   # DNS resolution
   nslookup api.juicebox.ai

   # Connectivity
   curl -v --connect-timeout 5 https://api.juicebox.ai/v1/health
   ```

2. **Check Load**
   - Review query complexity
   - Check for unusually large requests

3. **Increase Timeout**
   ```bash
   kubectl set env deployment/juicebox-integration JUICEBOX_TIMEOUT=60000
   ```

4. **Implement Circuit Breaker**
   - Enable circuit breaker if repeated timeouts
   - Serve cached/fallback data
```

## Incident Communication Template

```markdown
## Incident Report Template

**Incident ID:** INC-YYYY-MM-DD-XXX
**Status:** Investigating | Identified | Monitoring | Resolved
**Severity:** P1 | P2 | P3 | P4
**Start Time:** YYYY-MM-DD HH:MM UTC
**End Time:** (when resolved)

### Summary
[Brief description of the incident]

### Impact
- Users affected: [number/percentage]
- Features affected: [list]
- Duration: [time]

### Timeline
- HH:MM - Incident detected
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Mitigation applied
- HH:MM - Incident resolved

### Root Cause
[Description of what caused the incident]

### Resolution
[What was done to fix it]

### Action Items
- [ ] Action 1 (Owner, Due Date)
- [ ] Action 2 (Owner, Due Date)
```

## On-Call Checklist

```markdown
## On-Call Handoff Checklist

### Before Shift
- [ ] Access to all monitoring dashboards
- [ ] VPN/access to production systems
- [ ] Runbook bookmarked
- [ ] Escalation contacts available

### During Shift
- [ ] Check dashboards every 30 min
- [ ] Respond to alerts within SLA
- [ ] Document all incidents
- [ ] Escalate P1/P2 immediately

### End of Shift
- [ ] Handoff open incidents
- [ ] Update incident log
- [ ] Brief incoming on-call
```

## Output
- Diagnostic scripts
- Response procedures
- Communication templates
- On-call checklists

## Resources
- [Juicebox Status](https://status.juicebox.ai)
- [Support Portal](https://juicebox.ai/support)

## Next Steps
After incident, see `juicebox-data-handling` for data management.