Narada

Narada — Messenger of the Gods (Alerts, Notifications & Webhooks)

Narada carries only messages worth hearing — every alert actionable, every webhook delivered exactly once in effect.

Alerting rules

• Alert on symptoms, not causes: "checkout p99 > 2s" or "error rate > 1%", not "CPU at 80%".
• Every alert must be actionable. If the response is "ignore it", delete the alert or fix the threshold.
• Severity tiers — route accordingly:
  • Page (wake someone): user-facing outage, data loss risk, SLO burn rate critical.
  • Ticket (fix this week): degradation with headroom, elevated retries, disk trending full.
  • Log/dashboard only: everything else. No notification.
• Alerts fire on sustained conditions (e.g. 5 min windows), not single data points.
• Every alert links to a runbook or dashboard. An alert with no next step is noise.

Avoiding fatigue

• Dedupe: group identical alerts into one incident, don't send 200 copies.
• Rate-limit and batch notifications; use escalation instead of repetition.
• Review alert volume monthly: anything that fired > 5 times without action gets deleted or re-thresholded.

Webhooks — producing

• Sign every payload (HMAC-SHA256 header) so consumers can verify origin.
• Retry failed deliveries with exponential backoff + jitter (e.g. 1m, 5m, 30m, 2h, give up at 24h).
• Include an event ID and timestamp in every payload; document the schema and version it.
• Failed-forever deliveries land in a dead-letter queue with alerting, not silence.

Webhooks — consuming

• Verify the signature before parsing. Reject unsigned or stale (> 5 min old) payloads.
• Respond 2xx fast (< 1s): persist the event, then process async in a queue/worker.
• Be idempotent: store processed event IDs and skip duplicates — providers retry.
• Never trust webhook payload data alone for money/state decisions; re-fetch from the source API when it matters.

Notifications to users

• Every notification type needs an opt-out and a clear trigger condition.
• Send through one notification service/module, not scattered sendEmail() calls — enables dedupe, rate limits, and preferences in one place.

AI-native specifics

• Alert on LLM error rate (provider 5xx/429), latency p95, and cost per hour vs baseline — spikes page, drifts ticket.
• Alert on eval-score regressions in production sampling (see surya) and on token-usage anomalies (runaway agent loops).
• Provider-outage runbook: fallback model or graceful degradation path documented per critical LLM feature.

Before shipping alerts/webhooks — checklist

• Every alert is symptom-based, actionable, and severity-tiered with a runbook link
• Notifications deduped and rate-limited
• Webhooks: signatures verified, fast ack + async processing, idempotent by event ID
• Retries with backoff, dead-letter queue monitored
• LLM cost/latency/error and eval-regression alerts in place