gabrielmoreira Evernote Incident Runbook
Overview
Step-by-step procedures for responding to Evernote integration incidents including API outages, rate limit escalations, authentication failures, data sync issues, and quota exhaustion.
Prerequisites
- Access to monitoring dashboards and production logs
- Production Evernote API credentials
- Communication channels for escalation (Slack, PagerDuty)
Instructions
Incident Classification
| Severity | Symptoms | Response Time |
|---|---|---|
| P1 - Critical | All Evernote API calls failing, data loss risk | 15 minutes |
| P2 - High | Persistent rate limits, auth failures for multiple users | 1 hour |
| P3 - Medium | Intermittent errors, degraded sync performance | 4 hours |
| P4 - Low | Single user issues, non-critical feature affected | Next business day |
Step 1: Triage
Check Evernote's status page first. If Evernote is down, activate the circuit breaker and wait.
# Check Evernote service status
curl -sf https://status.evernote.com/api/v2/status.json | jq '.status'
# Check your API connectivity
curl -sf -H "Authorization: Bearer $EVERNOTE_TOKEN" \
https://www.evernote.com/shard/s1/notestore | head -20
# Check error rate in logs (last 15 min)
grep -c 'EDAMSystemException' /var/log/evernote-app.log
Step 2: Rate Limit Escalation
If rate limits are persistent: reduce API call frequency, increase delays between batch operations, and contact Evernote developer support for a rate limit increase.
Step 3: Authentication Failure
For auth failures: verify tokens are not expired (edam_expires), check that production credentials match the production endpoint (sandbox: false), and test with a fresh Developer Token to isolate the issue.
Step 4: Sync Failure Recovery
For sync issues: compare local USN with server USN via getSyncState(). If gap is too large, reset to full sync from USN 0. Verify data integrity after re-sync.
Step 5: Mitigation Strategies
- Circuit breaker: Disable Evernote API calls after N consecutive failures. Retry after cooldown period.
- Graceful degradation: Serve cached data when API is unavailable. Queue writes for retry.
- Failover: Switch to polling-based sync if webhooks stop arriving.
Post-Incident
- Document root cause and timeline
- Update runbook with new failure modes discovered
- Adjust alert thresholds if false positive or missed detection
- Review and improve circuit breaker settings
For the complete diagnostic scripts, mitigation implementations, and communication templates, see Implementation Guide.
Output
- Incident severity classification table
- Triage diagnostic commands for quick assessment
- Rate limit, auth, and sync failure response procedures
- Circuit breaker and graceful degradation patterns
- Post-incident review checklist
Error Handling
| Incident Type | Diagnostic | Mitigation |
|---|---|---|
| API outage | Check status.evernote.com |
Activate circuit breaker, serve cached data |
| Rate limit storm | Check evernote_rate_limits_total metric |
Reduce batch sizes, increase delays |
| Mass auth failure | Verify token expiration dates in DB | Trigger re-auth flow for affected users |
| Sync data loss | Compare local vs server note counts | Full re-sync from USN 0 |
Resources
Next Steps
For data handling best practices, see evernote-data-handling.
Examples
API outage response: Alert fires, on-call checks status page, confirms Evernote outage, activates circuit breaker, posts status update to internal Slack, monitors for recovery, then gradually re-enables API calls.
Rate limit recovery: Persistent RATE_LIMIT_REACHED errors detected. Reduce batch size from 100 to 10, increase delay to 500ms, clear the request queue, and contact Evernote support if limits continue after 1 hour.