# Alert > (forwward) Bootstraps production error alerting — wires app errors to Slack and/or Sentry through a single reporting seam, deduped notifications, and secret/flag/deploy plumbing done in the right order. Triggers on "add alerting", "wire up Slack alerts", "add Sentry", "we need to know when prod breaks", or setting up observability for a new service. - Skill: `iankiku/alert` (Agent Skill) - Install (CLI): `npx skillmds@latest add iankiku/alert` - Raw SKILL.md: https://api.skillmd.com/api/skills/iankiku/alert/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: iankiku (https://skillmd.com/u/iankiku) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/iankiku/alert --- # Alert — Error → Slack/Sentry Bootstrap Make a service tell you when it breaks in production, without a silent no-op. The pattern is the same across stacks: **one reporting seam → fan out to logging + Sentry + Slack → dedupe → gate behind a flag → seed the secret and deploy in the correct order.** Most rollouts get bitten by the *plumbing*, not the code — this front-loads those traps. **North star:** every unhandled error and every swallowed-but-important event reaches a human channel, exactly once per incident, with enough context to act — verified live, not assumed. ## Step 0 — Detect the stack and delivery path - **Language/runtime** — long-running server, serverless function, background worker? Determines where the reporter seam and flush hooks go. - **Where secrets live** — a cloud secret manager, CI provider secrets, or plain env vars? - **How config flags are set** — a config service, a YAML file, an env var, a feature-flag provider? - **Deploy mechanism** — does a fresh deploy clone the *currently running* config forward (common on container platforms), or rebuild config from source every time? This decides whether a new secret wires itself in automatically or needs a one-time bootstrap (see Step 4). ## Step 1 — One reporting seam, not scattered webhook calls Create a single module the whole app calls through, with two entry points: - `report(error, context)` → logging + Sentry + Slack. For unhandled/handled exceptions. - `notify(message, context)` → Slack only. For "this is fine but a human should know" events (e.g. a webhook you ack with 200 but couldn't fully process). Requirements baked into the seam: - **Dedupe** — collapse identical errors within a window (~5 min) by a stable key (type + message + call-site), so one bad deploy doesn't post thousands of Slack lines. - **Never throw from the reporter** — an alerting failure must not break the request path. Wrap the Slack/Sentry calls; on failure, log and move on. - **Context every time** — environment, service, request/trace id, and a short human summary. - **Server-side only** — a Slack webhook URL is a credential (see Step 2). It runs on the server; never ship it to a browser/client bundle. Wire the seam at the real failure points, not just a global handler: - Global exception handler / middleware → `report`. - **Background workers** — the silent killer is a `try/except` around the poll loop that swallows everything. Route that catch to `report`, add a shutdown-signal flush so queued alerts drain before exit, and a one-line recovery notice when it comes back. - **Webhook endpoints that ack with 200** even on partial failure → `notify` on the swallowed branch. - Frontend error boundary (if any) → Sentry's error hook. ## Step 2 — Secrets: the Slack webhook URL *is* the credential - A Slack incoming webhook URL carries no separate token — possessing the URL is the ability to post. Treat it as a secret: store it in the secret manager, server-only, never logged, never committed. - **A Sentry DSN is public by design** — safe as a plain env var / client config. Don't over-protect it. - Convention that scales to multi-env: one secret per environment (e.g. a dev webhook and a prod webhook), and the app selects by its own environment setting. Provision both in every stack so the service never fails to start on a missing one. ### ⚠️ Trap A — seed the secret as a plain string, not JSON-wrapped A generic secret-seeding helper may store `{"KEY":"value"}` while the app injects the bare secret value as the env var. Result: the env var holds a JSON blob, not a URL, the post silently fails, and you get a 200 with **no Slack message**. Seed the raw value if that's what the runtime expects, and verify what's actually stored *and* what the running instance received — don't trust "seed succeeded." ### ⚠️ Trap B — an empty secret can crash container startup On several container platforms, a secret with an empty value throws an initialization error and the task never starts. Provision each alert secret with a placeholder value at create time (so a fresh stack boots), then overwrite with the real URL — and make sure your infra tooling won't fight the manual seed on the next apply. ## Step 3 — Gate behind a flag (off by default), then enable deliberately - Put delivery behind a config flag, default **false**. Ship the code dark, enable per-environment once the secret is verified. - If the flag lives in a YAML file, watch indentation — a wrong-indent edit can silently no-op. Probe the *deployed* value after applying; don't assume the file edit took. - Correct order to go live: **seed secret → set flag → deploy (re-hydrate config)**. Doing config before secret, or forgetting the redeploy, yields a "configured but dead" alerting path. ## Step 4 — ⚠️ Trap C: a clone-forward deploy won't pick up a new secret on its own If your deploy pipeline clones the currently-running config/task definition forward and treats it as immutable outside CI, adding a new secret in IaC or a normal deploy can fail to wire it in — the running instance doesn't have it, so the clone doesn't either. Bootstrap it once: 1. Register a new revision that includes the new secret entry (+ env mapping). 2. Seed the secret value (Step 2). 3. Force a new deployment against that revision. 4. From then on, deploys clone *that* revision forward and it sticks. This single trap is the most common reason "I added Slack alerts and nothing happens in prod." ## Step 5 — Verify live - Trigger a real error or send a real test event through the actual ingress (not a unit test). Confirm the message lands in the channel for each environment you enabled. - Confirm the running instance shows the expected revision, the secret resolves to a URL (not JSON, not a placeholder), and the flag reads true in the *deployed* config. - Re-trigger to confirm dedupe collapses repeats. - For workers: kill/restart to confirm the shutdown flush and recovery notice fire. ## Per-environment go-live checklist ``` [ ] Reporting seam in place; report() + notify() wired at all failure points (incl. worker catch + 200-ack webhooks) [ ] Slack webhook secret seeded plain (verified server-side), per-env, never client-exposed [ ] Secret placeholder set so a fresh instance boots; infra tooling won't overwrite the manual seed [ ] Sentry DSN wired (plain env ok); error boundary hooked up [ ] Flag default false; enabled deliberately per env; deployed value probed, not assumed [ ] Clone-forward deploys bootstrapped once with the new secret entry [ ] Live test fired through real ingress → message seen in channel, per env [ ] Dedupe + shutdown flush + recovery notice verified ``` ## Anti-patterns - Scattering ad-hoc Slack post calls through the codebase instead of one reporter seam. - Seeding the webhook JSON-wrapped when the app wants the raw URL → silent no-op. - Enabling the flag before the secret is verified → "configured but dead." - Assuming a deploy picks up a new secret on a clone-forward pipeline (Trap C). - Calling it done after a green deploy without seeing a real alert land in the channel. - Letting a worker's poll-loop `try/except` swallow errors with no `report()`.