Webhooks

Webhooks — the inbound front door that survives retries and forgeries

You are building the side of a webhook that receives. Some external system pushes an HTTP POST at your endpoint; your job is to prove it is real, refuse to act on it twice, and answer fast. That is the whole mandate.

This skill is provider-agnostic — it teaches the mechanics every webhook source shares, not any one vendor's event catalog — and it ends at "enqueued." What happens to the event afterward is a different job:

• The Stripe-specific scheme (Stripe-Signature, t=,v1=, stripe listen, the event model) → ../stripe/SKILL.md.
• Email bounce/complaint webhooks where the point is suppression state → ../email-connector/SKILL.md.
• The outbound client that calls someone else's API → api-connector-builder.
• Multi-step orchestration after the event lands → automation-flows.
• Broker/queue tuning (BullMQ concurrency, DLQ ops) → redis.
• Constant-time compare, secret handling, supply-chain hygiene → ../secure-coding/SKILL.md.

The pipeline

These five stages run in exactly this order. Each one exists to protect the one after it; reorder them and you either trust forged data or waste work.

Delivery is at-least-once, never exactly-once. Every major provider retries on non-2xx or timeout, so duplicate deliveries are normal traffic and idempotency is required, not a nice-to-have.

Verify the signature

Verify over the raw bytes, never over re-serialized JSON. Re-serializing a parsed object reorders keys and normalizes whitespace, so the HMAC you compute no longer matches the one the sender computed. This is the single most common cause of "signature fails even though the secret is right."

// Bad — body was parsed, so the bytes are gone. HMAC will never match.
app.post("/webhooks", express.json(), (req, res) => {
  const expected = sign(JSON.stringify(req.body)); // reordered, re-spaced
});

// Good — keep the raw buffer for the HMAC; parse only after verifying.
app.post("/webhooks", express.raw({ type: "*/*" }), (req, res) => {
  const raw = req.body; // a Buffer, the exact bytes received
  if (!verify(raw, req.headers)) return res.sendStatus(400);
  const event = JSON.parse(raw.toString("utf8"));
});

Compare signatures in constant time. A normal ===/== short-circuits on the first differing byte, which leaks how much of the signature an attacker guessed correctly. Use crypto.timingSafeEqual (Node) or hmac.compare_digest (Python). The buffers must be equal length, so guard that first.

The cross-vendor Standard Webhooks spec (adopted by Svix, Clerk, Resend, Brex and others) defines the scheme this skill defaults to: headers webhook-id, webhook-timestamp, webhook-signature; the signature is base64(HMAC-SHA256(secret, "{id}.{timestamp}.{body}")), version-prefixed as v1,<sig>, and the header may carry space-separated signatures so a secret can be rotated without dropped deliveries — accept the request if any match.

import crypto from "node:crypto";

function verify(raw, headers) {
  const id = headers["webhook-id"];
  const ts = headers["webhook-timestamp"];
  const secret = process.env.WEBHOOK_SECRET; // never a literal
  const signed = `${id}.${ts}.${raw.toString("utf8")}`;
  const expected = crypto
    .createHmac("sha256", Buffer.from(secret, "base64"))
    .update(signed)
    .digest("base64");
  // header is "v1,<sig> v1,<sig2>"; accept if any matches (secret rotation)
  return String(headers["webhook-signature"] || "")
    .split(" ")
    .map((part) => part.split(",")[1])
    .some((sig) => sameLength(sig, expected) &&
      crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected)));
}
const sameLength = (a = "", b = "") => a.length === b.length;

import hmac, hashlib, base64, os

def verify(raw: bytes, headers) -> bool:
    secret = base64.b64decode(os.environ["WEBHOOK_SECRET"])  # never a literal
    signed = f'{headers["webhook-id"]}.{headers["webhook-timestamp"]}.'.encode() + raw
    expected = base64.b64encode(hmac.new(secret, signed, hashlib.sha256).digest()).decode()
    sent = headers.get("webhook-signature", "")
    for part in sent.split(" "):          # space-separated for rotation
        sig = part.split(",", 1)[-1]
        if hmac.compare_digest(sig, expected):  # constant time
            return True
    return False

Per-provider header formats (Stripe t=,v1=, GitHub X-Hub-Signature-256, Shopify base64 HMAC, Slack v0=, Svix) all map onto the same primitive — mapping table and per-language verify primitives in references/signature-schemes.md.

Timestamp & replay

A valid signature does not stop a replay: an attacker who captures one delivery can re-send the exact bytes, signature intact. Bound it with a timestamp.

Reject any event whose webhook-timestamp is more than ~5 minutes from now. That five-minute window is the de-facto industry value (Benchling and others recommend it). The Standard Webhooks spec mandates that you perform a tolerance check but does not fix the number — pick a small one.

const skewSeconds = Math.abs(Date.now() / 1000 - Number(ts));
if (skewSeconds > 300) return res.sendStatus(400); // outside the 5-min window

The timestamp window bounds how long a captured payload is replayable; the idempotency check (next section) stops exact replays that arrive inside the window. You need both — neither alone is enough.

Idempotency

The idempotency key is the provider's stable event id (webhook-id, or for Stripe event.id) — never a hash of the body or your own generated id. Mark it seen before doing the work, so a retry that arrives mid-processing is caught.

Two stores, same idea:

The dedupe TTL must outlive the provider's entire retry schedule. Retries are exponential and capped — a provider may redeliver hours or days later. If your key expires first, that late retry looks brand-new and you process twice.

// Insert-then-process: the UNIQUE insert is the gate. Do work only if we won it.
try {
  await db.query("INSERT INTO seen_events (id) VALUES ($1)", [id]);
} catch (e) {
  if (e.code === "23505") return res.sendStatus(200); // already seen → ack, skip
  throw e;
}
await queue.add("process", { id, event });
res.sendStatus(200);

Prefer insert-then-process over process-then-mark. Marking after the work leaves a window where two concurrent deliveries both pass the "have I seen it?" check and both run. Let the UNIQUE constraint (or NX) be the lock.

Fast-ack & queue

Heavy synchronous work inside the handler causes provider timeouts and a storm of retries. Return 2xx as soon as the event is durably stored or enqueued, then do the real work in a background worker. The status code is a contract with the sender:

Once work is in the queue, handle failures there: bounded retries with backoff, then route exhausted jobs to a dead-letter queue / poison shelf for manual inspection. Do not let a poison event loop forever or block the queue head. Queue/broker tuning itself (concurrency, DLQ wiring) is redis's job.

Framework raw-body recipes

Capturing the raw body is framework-specific and is the usual culprit behind "mismatch though the secret is correct." One line each; full recipes in references/framework-raw-body.md:

• Express — mount express.raw({ type: "*/*" }) on the webhook route only; do not let a global express.json() consume it first.
• Next.js App Router — read await req.text() (or req.arrayBuffer()) in the route handler; the App Router does not auto-parse, but await req.json() discards the raw bytes, so verify off the text.
• FastAPI / Starlette — await request.body() for the raw bytes; do not type the handler param as a Pydantic model (that parses for you).
• Hono — await c.req.text() before any c.req.json().
• Serverless (Lambda/Vercel/Cloudflare) — disable body parsing / read the raw stream; base64-decode if the platform wraps the body.

Anti-patterns

Verify your handler

scripts/verify.sh is a read-only heuristic linter. Run it from the root of the project that contains your handler; it scans candidate files and prints PASS/WARN/FAIL per invariant (raw-body verify, constant-time compare, timestamp check, secret-from-env, dedupe-on-event-id). A WARN means "I could not find evidence," which on a clean tree is expected and exits 0.