Alert

Alert — Error → Slack/Sentry Bootstrap

Make a service tell you when it breaks in production, without a silent no-op. The pattern is the same across stacks: one reporting seam → fan out to logging + Sentry + Slack → dedupe → gate behind a flag → seed the secret and deploy in the correct order. Most rollouts get bitten by the plumbing, not the code — this front-loads those traps.

North star: every unhandled error and every swallowed-but-important event reaches a human channel, exactly once per incident, with enough context to act — verified live, not assumed.

Step 0 — Detect the stack and delivery path

• Language/runtime — long-running server, serverless function, background worker? Determines where the reporter seam and flush hooks go.
• Where secrets live — a cloud secret manager, CI provider secrets, or plain env vars?
• How config flags are set — a config service, a YAML file, an env var, a feature-flag provider?
• Deploy mechanism — does a fresh deploy clone the currently running config forward (common on container platforms), or rebuild config from source every time? This decides whether a new secret wires itself in automatically or needs a one-time bootstrap (see Step 4).

Step 1 — One reporting seam, not scattered webhook calls

Create a single module the whole app calls through, with two entry points:

• report(error, context) → logging + Sentry + Slack. For unhandled/handled exceptions.
• notify(message, context) → Slack only. For "this is fine but a human should know" events (e.g. a webhook you ack with 200 but couldn't fully process).

Requirements baked into the seam:

• Dedupe — collapse identical errors within a window (~5 min) by a stable key (type + message + call-site), so one bad deploy doesn't post thousands of Slack lines.
• Never throw from the reporter — an alerting failure must not break the request path. Wrap the Slack/Sentry calls; on failure, log and move on.
• Context every time — environment, service, request/trace id, and a short human summary.
• Server-side only — a Slack webhook URL is a credential (see Step 2). It runs on the server; never ship it to a browser/client bundle.

Wire the seam at the real failure points, not just a global handler:

• Global exception handler / middleware → report.
• Background workers — the silent killer is a try/except around the poll loop that swallows everything. Route that catch to report, add a shutdown-signal flush so queued alerts drain before exit, and a one-line recovery notice when it comes back.
• Webhook endpoints that ack with 200 even on partial failure → notify on the swallowed branch.
• Frontend error boundary (if any) → Sentry's error hook.

Step 2 — Secrets: the Slack webhook URL is the credential

• A Slack incoming webhook URL carries no separate token — possessing the URL is the ability to post. Treat it as a secret: store it in the secret manager, server-only, never logged, never committed.
• A Sentry DSN is public by design — safe as a plain env var / client config. Don't over-protect it.
• Convention that scales to multi-env: one secret per environment (e.g. a dev webhook and a prod webhook), and the app selects by its own environment setting. Provision both in every stack so the service never fails to start on a missing one.

⚠️ Trap A — seed the secret as a plain string, not JSON-wrapped

A generic secret-seeding helper may store {"KEY":"value"} while the app injects the bare secret value as the env var. Result: the env var holds a JSON blob, not a URL, the post silently fails, and you get a 200 with no Slack message. Seed the raw value if that's what the runtime expects, and verify what's actually stored and what the running instance received — don't trust "seed succeeded."

⚠️ Trap B — an empty secret can crash container startup

On several container platforms, a secret with an empty value throws an initialization error and the task never starts. Provision each alert secret with a placeholder value at create time (so a fresh stack boots), then overwrite with the real URL — and make sure your infra tooling won't fight the manual seed on the next apply.

Step 3 — Gate behind a flag (off by default), then enable deliberately

• Put delivery behind a config flag, default false. Ship the code dark, enable per-environment once the secret is verified.
• If the flag lives in a YAML file, watch indentation — a wrong-indent edit can silently no-op. Probe the deployed value after applying; don't assume the file edit took.
• Correct order to go live: seed secret → set flag → deploy (re-hydrate config). Doing config before secret, or forgetting the redeploy, yields a "configured but dead" alerting path.

Step 4 — ⚠️ Trap C: a clone-forward deploy won't pick up a new secret on its own

If your deploy pipeline clones the currently-running config/task definition forward and treats it as immutable outside CI, adding a new secret in IaC or a normal deploy can fail to wire it in — the running instance doesn't have it, so the clone doesn't either. Bootstrap it once:

1. Register a new revision that includes the new secret entry (+ env mapping).
2. Seed the secret value (Step 2).
3. Force a new deployment against that revision.
4. From then on, deploys clone that revision forward and it sticks.

This single trap is the most common reason "I added Slack alerts and nothing happens in prod."

Step 5 — Verify live

• Trigger a real error or send a real test event through the actual ingress (not a unit test). Confirm the message lands in the channel for each environment you enabled.
• Confirm the running instance shows the expected revision, the secret resolves to a URL (not JSON, not a placeholder), and the flag reads true in the deployed config.
• Re-trigger to confirm dedupe collapses repeats.
• For workers: kill/restart to confirm the shutdown flush and recovery notice fire.

Per-environment go-live checklist

[ ] Reporting seam in place; report() + notify() wired at all failure points (incl. worker catch + 200-ack webhooks)
[ ] Slack webhook secret seeded plain (verified server-side), per-env, never client-exposed
[ ] Secret placeholder set so a fresh instance boots; infra tooling won't overwrite the manual seed
[ ] Sentry DSN wired (plain env ok); error boundary hooked up
[ ] Flag default false; enabled deliberately per env; deployed value probed, not assumed
[ ] Clone-forward deploys bootstrapped once with the new secret entry
[ ] Live test fired through real ingress → message seen in channel, per env
[ ] Dedupe + shutdown flush + recovery notice verified

Anti-patterns

• Scattering ad-hoc Slack post calls through the codebase instead of one reporter seam.
• Seeding the webhook JSON-wrapped when the app wants the raw URL → silent no-op.
• Enabling the flag before the secret is verified → "configured but dead."
• Assuming a deploy picks up a new secret on a clone-forward pipeline (Trap C).
• Calling it done after a green deploy without seeing a real alert land in the channel.
• Letting a worker's poll-loop try/except swallow errors with no report().