Dhanvantari

Dhanvantari — God of Medicine (Incidents & Hotfixes)

Dhanvantari heals production: stop the bleeding first, understand the disease after.

Declare and organize

• Declare severity immediately: SEV1 (users down / data at risk), SEV2 (major feature broken, workaround exists), SEV3 (degraded, low impact).
• Name one incident lead. The lead coordinates and communicates; others debug. No lead = chaos.
• Open a dedicated channel/thread per incident; all findings and actions go there, timestamped.

Mitigate first, diagnose later

• Under pressure, rollback beats forward-fix. If the incident started after a deploy, roll back that deploy first and investigate after.
• Other fast mitigations before root-causing: feature flag off, scale up, failover, rate-limit the offending client.
• Never debug in prod without a second person aware — one person watching prevents a bad incident becoming a worse one.
• Communicate status to stakeholders every 30 minutes during SEV1/SEV2, even if the update is "still investigating". Silence reads as abandonment.

Hotfix flow

• Branch hotfix/<desc> from main, minimal diff — fix only the incident, no drive-by refactors.
• Expedited review: one reviewer, synchronous, minutes not hours. CI still runs (see brahma).
• Deploy, verify the fix in prod with metrics/logs, then backport to any active release branches.

After the fire

• Blameless postmortem within 48 hours for SEV1/SEV2: timeline, root cause, what went well, what didn't, action items — each with an owner and a due date. No owner = won't happen.
• "Human error" is never a root cause; ask what guardrail was missing.
• Keep a runbook per service: how to check health, restart, roll back, common failure modes, escalation contacts. Update it after every incident that revealed a gap.

AI-native specifics

• LLM providers go down. Every AI feature needs a degradation plan: fallback model (different provider), cached/canned responses, or graceful "AI temporarily unavailable" — never a hard crash.
• Wrap AI features in kill-switch feature flags so a misbehaving model, prompt regression, or runaway cost can be turned off in seconds without a deploy.
• During provider incidents, watch retry storms: exponential backoff with jitter, circuit breaker so retries don't amplify the outage or the bill (see lakshmi).

During any incident — checklist

• Severity declared, incident lead named
• Mitigation attempted before deep diagnosis (rollback / flag off)
• Stakeholder update every 30 min
• Hotfix is minimal diff from main, reviewed, backported
• Blameless postmortem within 48h, action items owned