Backend Webhooks

Backend Webhooks

Purpose

Securely receive, verify, retry, and deliver webhooks between services with guaranteed delivery and tamper-proof signatures. Webhooks are the standard mechanism for asynchronous service-to-service event notification.

Agent Protocol

Trigger

Exact user phrases: "webhook", "webhook receiver", "webhook delivery", "signature verification", "HMAC", "event delivery", "outgoing webhook", "incoming webhook", "retry webhook", "webhook endpoint".

Input Context

• Direction: receiving incoming webhooks or sending outgoing webhooks.
• Security requirements (signing, IP allowlisting).
• Expected volume and payload size.
• Event types and schemas.

Output Artifact

Webhook handler code or configuration. No file unless requested.

Response Format

Direction: {incoming|outgoing}
Auth: {HMAC|Basic|OAuth|None}
Retry: {strategy}
Delivery Guarantee: {at-least-once|exactly-once}

Completion Criteria

• Signature verification for incoming webhooks.
• Payload signing for outgoing webhooks.
• Retry with exponential backoff for failed deliveries.
• Dead-letter queue for permanently failed deliveries.
• At-least-once delivery guaranteed.
• Idempotency handling for duplicate deliveries.

Max Response Length

4 lines per webhook. 20 lines for full implementation.

Architecture Decision Tree

Should I Use Webhooks or Polling?

Does the consumer need near-real-time notification?
  ├── Yes → Webhooks (push-based, lower latency)
  └── No → Is the consumer behind a firewall?
            ├── Yes → Polling (consumer pulls when ready)
            └── No → Does the consumer need guaranteed delivery?
                      ├── Yes → Webhooks + retry + DLQ
                      └── No → Webhooks (best-effort)

Outgoing vs Incoming Webhook Design

Are you sending events to external systems?
  ├── Yes → Outgoing webhooks:
  │   ├── Subscription management (CRUD endpoints)
  │   ├── Payload signing (HMAC-SHA256)
  │   ├── Retry with backoff (5 attempts)
  │   ├── Dead letter queue (after max retries)
  │   ├── Rate limiting per subscription
  │   └── Consumer-managed endpoint configuration
  └── No → Incoming webhooks:
      ├── Signature verification (timing-safe)
  │   ├── Timestamp validation (tolerance window)
  │   ├── Payload validation (schema check)
  │   ├── Idempotency handling (X-Webhook-ID)
      └── Fast acknowledgment (200 before processing)

Authentication Decision

What level of security does the integration need?
  ├── HMAC signature → Most common, simple, tamper-proof
  │   ├── Shared secret known to both parties
  │   ├── Payload + timestamp signed with SHA256
  │   └── Use timing-safe comparison for verification
  ├── OAuth2/Bearer token → Standard for API-first services
  │   ├── Provider issues access token
  │   └── Consumer validates token on each webhook
  ├── IP allowlisting → Additional layer, not standalone
  │   └── Only accept from known IP ranges
  └── Mutual TLS → Highest security, operational overhead
      └── Both parties present certificates

Workflow

Step 1: Define Event Types

events:
  order.created:
    description: Order has been placed
    schema: OrderCreatedEvent
    version: 1
  order.fulfilled:
    description: Order has been shipped
    schema: OrderFulfilledEvent
    version: 1
  order.cancelled:
    description: Order was cancelled
    schema: OrderCancelledEvent
    version: 1

Step 2: Sign Outgoing Payloads

function signPayload(payload, secret) {
  const timestamp = Math.floor(Date.now() / 1000);
  const payloadString = JSON.stringify(payload);
  const signature = crypto
    .createHmac('sha256', secret)
    .update(`${timestamp}.${payloadString}`)
    .digest('hex');
  return { signature, timestamp };
}

async function deliverWebhook(subscription, event) {
  const { signature, timestamp } = signPayload(event, subscription.secret);
  const headers = {
    'Content-Type': 'application/json',
    'X-Webhook-Signature': `sha256=${signature}`,
    'X-Webhook-Timestamp': timestamp,
    'X-Webhook-ID': event.id,
    'X-Webhook-Event': event.type,
  };
  return fetch(subscription.url, {
    method: 'POST',
    headers,
    body: JSON.stringify(event),
  });
}

Step 3: Verify Incoming Signatures

function verifyWebhook(payload, headers, secret, toleranceMs = 300000) {
  const signature = headers['x-webhook-signature'];
  const timestamp = parseInt(headers['x-webhook-timestamp'], 10);
  const eventId = headers['x-webhook-id'];

  // 1. Validate timestamp is within tolerance window
  const age = Date.now() - timestamp * 1000;
  if (Math.abs(age) > toleranceMs) {
    throw new Error(`Webhook timestamp expired: age=${age}ms > tolerance=${toleranceMs}ms`);
  }

  // 2. Verify signature using timing-safe comparison
  const payloadString = JSON.stringify(payload);
  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(`${timestamp}.${payloadString}`)
    .digest('hex');

  // Extract scheme from signature header
  const [scheme, actualSig] = signature.split('=');
  if (scheme !== 'sha256') throw new Error(`Unknown signature scheme: ${scheme}`);

  // Timing-safe comparison
  const actualBuf = Buffer.from(actualSig);
  const expectedBuf = Buffer.from(expectedSig);
  if (actualBuf.length !== expectedBuf.length || !crypto.timingSafeEqual(actualBuf, expectedBuf)) {
    throw new Error('Webhook signature invalid');
  }

  // 3. Return event metadata
  return { eventId, timestamp, verified: true };
}

Step 4: Implement Delivery with Retry

interface DeliveryResult {
  success: boolean;
  statusCode?: number;
  error?: string;
  attempt: number;
}

class WebhookDeliveryService {
  private maxRetries = 5;
  private baseDelay = 1000;

  async deliver(subscription: Subscription, event: WebhookEvent): Promise<DeliveryResult> {
    let lastError: string | undefined;

    for (let attempt = 0; attempt < this.maxRetries; attempt++) {
      try {
        const response = await this.send(subscription, event, attempt);
        if (response.ok) {
          await this.recordSuccess(subscription, event, attempt);
          return { success: true, statusCode: response.status, attempt };
        }
        // Non-2xx — retry if retryable
        if (response.status >= 400 && response.status < 500) {
          // Client error — don't retry
          await this.recordFailure(subscription, event, response.status, 'Client error');
          return { success: false, statusCode: response.status, attempt };
        }
        lastError = `HTTP ${response.status}`;
      } catch (error) {
        lastError = error.message;
      }

      // Exponential backoff with jitter
      const delay = Math.min(
        this.baseDelay * Math.pow(2, attempt) + Math.random() * 1000,
        60000 // Max 60s
      );
      await sleep(delay);
    }

    // All retries exhausted — send to DLQ
    await this.deadLetter(subscription, event, lastError);
    await this.alertOncall(subscription, event, lastError);
    return { success: false, error: lastError, attempt: this.maxRetries };
  }
}

Step 5: Handle Duplicates (Idempotency)

// Consumers should deduplicate by X-Webhook-ID
async function handleIncomingWebhook(req, res) {
  const webhookId = req.headers['x-webhook-id'];

  // Check if already processed
  const processed = await idempotencyStore.get(webhookId);
  if (processed) {
    return res.status(200).json({ status: 'duplicate' });
  }

  // Process webhook
  await processEvent(req.body);

  // Mark as processed
  await idempotencyStore.set(webhookId, { processed: true }, { ttl: 86400 });

  res.status(200).json({ status: 'ok' });
}

Step 6: Subscription Management

interface WebhookSubscription {
  id: string;
  url: string;
  secret: string;
  events: string[];          // Event types to receive
  status: 'active' | 'disabled' | 'suspended';
  rateLimit: number;         // Max requests per minute
  retryConfig: {
    maxRetries: number;
    backoffBaseMs: number;
  };
  createdAt: Date;
  lastSuccessAt?: Date;
  lastFailureAt?: Date;
  consecutiveFailures: number;
}

// CRUD API for managing subscriptions
// POST   /api/webhooks/subscriptions    — Create subscription
// GET    /api/webhooks/subscriptions    — List subscriptions
// PUT    /api/webhooks/subscriptions/:id — Update subscription
// DELETE /api/webhooks/subscriptions/:id — Delete subscription
// POST   /api/webhooks/subscriptions/:id/rotate — Rotate secret

class SubscriptionManager {
  async createSubscription(input: CreateSubscriptionInput): Promise<WebhookSubscription> {
    // Validate URL is reachable
    await this.validateEndpoint(input.url);

    // Generate unique secret
    const secret = crypto.randomBytes(32).toString('hex');

    // Send test webhook
    await this.sendTestWebhook(input.url, secret);

    const subscription: WebhookSubscription = {
      id: crypto.randomUUID(),
      ...input,
      secret,
      status: 'active',
      rateLimit: input.rateLimit || 100,
      retryConfig: { maxRetries: 5, backoffBaseMs: 1000 },
      consecutiveFailures: 0,
      createdAt: new Date(),
    };

    await this.store.save(subscription);
    return subscription;
  }

  private async sendTestWebhook(url: string, secret: string): Promise<void> {
    const testPayload = { type: 'ping', data: { message: 'Webhook subscription test' } };
    const { signature, timestamp } = signPayload(testPayload, secret);
    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': `sha256=${signature}`,
        'X-Webhook-Timestamp': String(timestamp),
        'X-Webhook-ID': crypto.randomUUID(),
        'X-Webhook-Event': 'ping',
      },
      body: JSON.stringify(testPayload),
    });
    if (!response.ok) throw new Error(`Test webhook failed: HTTP ${response.status}`);
  }
}

Implementation Patterns

Webhook Server (Express)

import express from 'express';

const app = express();

// Raw body for signature verification
app.use(express.json({
  verify: (req, _, buf) => { req.rawBody = buf.toString(); },
}));

// Incoming webhook handler
app.post('/webhooks/:provider', async (req, res) => {
  // Acknowledge immediately
  res.status(202).json({ received: true });

  // Process asynchronously
  const provider = req.params.provider;
  const secret = getProviderSecret(provider);

  try {
    verifyWebhook(JSON.parse(req.rawBody), req.headers, secret);
    await processWebhookAsync(req.body);
  } catch (error) {
    logger.error('Webhook processing failed', { provider, error });
  }
});

Dead Letter Queue

class WebhookDeadLetterQueue {
  async sendToDLQ(subscription: Subscription, event: WebhookEvent, error: string): Promise<void> {
    await this.store.save({
      id: crypto.randomUUID(),
      subscriptionId: subscription.id,
      url: subscription.url,
      event,
      error,
      failedAt: new Date(),
      retryCount: 0,
    });
  }

  async retryDeadLetter(limit = 100): Promise<void> {
    const failed = await this.store.findUnprocessed(limit);
    for (const item of failed) {
      try {
        const subscription = await this.subscriptionManager.get(item.subscriptionId);
        await this.deliveryService.deliver(subscription, item.event);
        await this.store.markProcessed(item.id);
      } catch (error) {
        logger.error('DLQ retry failed', { id: item.id, error });
      }
    }
  }
}

Rate Limiting Outgoing Webhooks

class WebhookRateLimiter {
  private attempts = new Map<string, number[]>();

  async canDeliver(subscriptionId: string): Promise<boolean> {
    const now = Date.now();
    const window = 60000; // 1 minute
    const attempts = this.attempts.get(subscriptionId) || [];
    const recent = attempts.filter(t => now - t < window);

    const subscription = await this.subscriptionManager.get(subscriptionId);
    if (recent.length >= subscription.rateLimit) {
      logger.warn('Webhook rate limit exceeded', { subscriptionId });
      return false;
    }

    recent.push(now);
    this.attempts.set(subscriptionId, recent);
    return true;
  }
}

Production Considerations

Retry Strategy

After 5 retries: send to DLQ and alert.

Monitoring

Scale Considerations

Anti-Patterns

1. Unsigned payloads: Without HMAC, any attacker can forge webhooks. Always sign every payload.
2. No replay protection: Without timestamp validation, an attacker can replay a captured webhook. Always enforce a tolerance window.
3. Blocking on webhook processing: Receiving a webhook should return 200 immediately and process async. Blocking during processing causes timeouts and retries.
4. Non-atomic secret rotation: Updating the secret while a webhook is in flight causes verification failures. Support dual secrets during rotation.
5. No dead letter queue: Failed deliveries are lost forever. Always have a DLQ for retry-exhausted webhooks.
6. Ignoring idempotency: Webhook systems deliver at-least-once. Without idempotency handling, duplicates cause data corruption.
7. No consumer health monitoring: A continuously failing consumer should be auto-disabled to prevent resource waste.

Security

IP Allowlisting

const ALLOWED_IPS = [
  '54.123.45.0/24',
  '54.67.89.0/24',
];

function isAllowedIP(ip: string): boolean {
  return ALLOWED_IPS.some(cidr => ipInCIDR(ip, cidr));
}

Secret Rotation

async function rotateSecret(subscriptionId: string): Promise<void> {
  const newSecret = crypto.randomBytes(32).toString('hex');
  const subscription = await subscriptionStore.get(subscriptionId);

  // Dual-write period: accept both old and new signatures
  subscription.pendingSecret = newSecret;
  subscription.pendingSecretActivatedAt = Date.now() + 600000; // 10 min overlap
  await subscriptionStore.save(subscription);

  // After overlap period, remove old secret
  setTimeout(async () => {
    const updated = await subscriptionStore.get(subscriptionId);
    if (updated.pendingSecret === newSecret) {
      updated.secret = newSecret;
      updated.pendingSecret = null;
      updated.pendingSecretActivatedAt = null;
      await subscriptionStore.save(updated);
    }
  }, 600000);
}

Rules

• Always sign webhook payloads with HMAC-SHA256 — never send unsigned payloads.
• Include a timestamp in the signature and enforce a tolerance window (5 minutes max).
• Use timing-safe comparison for signature verification.
• Retry at least 5 times with exponential backoff before dead-lettering.
• Log all delivery attempts (success and failure) with timestamps.
• Rotate webhook secrets regularly.
• Allow consumers to unsubscribe — respect opt-out.
• Acknowledge incoming webhooks immediately (202), process asynchronously.
• Auto-disable consumers with > 10 consecutive failures.

References

• references/webhook-delivery.md — Webhook Delivery System
• references/webhook-monitoring.md — Webhook Monitoring and Reliability
• references/webhook-rate-limiting.md — Webhook Rate Limiting
• references/webhook-scaling.md — Webhook Scaling
• references/webhook-security.md — Webhook Security
• references/webhook-setup.md — Webhook Setup
• references/webhooks-architecture.md — Webhooks Architecture
• references/webhooks-delivery.md — Webhook Security

Handoff

No artifact produced unless requested. Next skill: api-versioning — manage version transitions for the webhook API. Carry forward: event types, webhook URL structure, signature format.