arbazkhan971 Webhook -- Design, Delivery & Security
Activate When
/godmode:webhook, "write a webhook", "webhook handler"- "webhook delivery", "callback URL"
- "verify webhook signature", "retry failed webhooks"
- "dead letter queue", "event notifications"
- When API needs to notify external systems
Workflow
Step 1: Discovery
# Detect webhook libraries and patterns
grep -r "svix\|standardwebhooks\|hmac\|createHmac" \
package.json src/ --include="*.ts" -l 2>/dev/null
grep -r "webhook" --include="*.sql" \
--include="*.prisma" -l 2>/dev/null
# Check queue integration
grep -r "BullMQ\|bull\|celery\|SQS\|amqplib" \
package.json 2>/dev/null
WEBHOOK DISCOVERY:
Direction: Inbound | Outbound | Both
Events: <list event types>
Volume: <events/sec, daily total>
Latency SLA: <max delay>
Delivery: At-least-once | Best-effort
Auth: HMAC-SHA256 | mTLS | bearer token
IF inbound only: skip delivery system (Step 4)
IF outbound only: skip inbound handler (Step 3)
IF volume > 1000/sec: add partitioned queue
IF volume < 10/min: simple async dispatch OK
Step 2: Event & Payload Design
NAMING: <resource>.<past_tense_verb>
e.g., order.created, payment.completed
ENVELOPE:
id: evt_<ULID> (unique, sortable)
type: resource.verb
api_version: 2025-01-15
created_at: ISO 8601 UTC
data: { object: {...}, previous_attributes: {} }
livemode: true|false
RULES:
Max payload: 64KB
Monetary values: cents (integer)
Timestamps: ISO 8601 UTC
NEVER include PII or secrets in payloads
IF payload > 64KB: send reference + fetch URL
WHEN previous state needed: include diff
Step 3: Inbound Webhook Handling
HANDLER PATTERN (strict order):
1. Verify HMAC-SHA256 signature
Use raw bytes, constant-time comparison
IF invalid: return 401 immediately
2. Check timestamp (5 min tolerance window)
IF stale (>300s drift): return 403
3. Check idempotency (event ID seen?)
IF duplicate: return 200 (already processed)
4. Parse payload, route to handler
5. Process in transaction
6. Return 200 within 5s timeout
IF heavy work: offload to queue
SSRF PREVENTION (inbound callback URLs):
Resolve DNS before connecting
Block: 127.0.0.0/8, 10.0.0.0/8,
172.16.0.0/12, 192.168.0.0/16,
169.254.0.0/16, ::1
Re-check on every retry
Step 4: Outbound Delivery System
PIPELINE:
Event Source -> Dispatcher -> Worker -> Endpoint
Delivery Log Retry Queue (exp backoff)
|
Dead Letter Queue
RETRY: 10 attempts over ~3 days
Delays: 0s, 10s, 30s, 1m, 5m, 15m, 1h, 4h, 8h, 24h
Jitter: random 0.5x-1.5x multiplier
IF 5xx, 408, 429, timeout, DNS fail: retry
IF 4xx (except 408/429): do NOT retry
WHEN Retry-After header present: honor it
CIRCUIT BREAKER (per endpoint):
CLOSED -> 5 consecutive failures -> OPEN
OPEN: queue events, alert owner
After 30 min -> HALF-OPEN (test 1 event)
IF 2 successes -> CLOSED
IF failure -> OPEN again
Auto-disable after 72 hours open
Step 5: Dead Letter Queue
Events land here after max retries (10)
or prolonged circuit break (>72h)
Retention: 30 days
API: replay single, replay bulk, export, purge
WHEN DLQ grows > 1000 entries: alert + analyze
Step 6: Subscription API
POST/GET /api/v1/webhooks Create/list
GET/PATCH /api/v1/webhooks/:id Get/update
DELETE /api/v1/webhooks/:id Delete
POST /api/v1/webhooks/:id/test Test ping
POST /api/v1/webhooks/:id/rotate Rotate secret
GET /api/v1/webhooks/:id/deliveries History
LIMITS:
HTTPS only (no HTTP in production)
Public IP only (SSRF block)
Max 100 event types per subscription
Max 20 subscriptions per account
Event filtering: ["*"], ["order.*"], specific
WHEN subscription created: send test ping
IF test ping fails: warn but allow save
Step 7: Secret Rotation
Dual-secret zero-downtime rotation:
Phase 1: sign with both old + new secret
Phase 2: grace period 24-72 hours
Phase 3: revoke old secret
Secret shown ONCE at creation.
Rotation cadence: 90 days recommended.
IF >180 days since rotation: warn in dashboard
Step 8: Database Schema
-- Core tables
webhook_subscriptions (
id, account_id, url, events JSONB,
secret_current, secret_previous,
circuit_state, failure_count,
disabled_at, created_at, updated_at
)
webhook_events (id, type, data JSONB, created_at)
webhook_deliveries (
id, event_id, subscription_id,
status, attempts, next_retry_at,
response_code, response_body_preview,
created_at
)
webhook_dead_letter (
delivery_id, reason, replayed_at
)
Indexes: subscription(account_id),
delivery(event_id, subscription_id),
delivery(status, next_retry_at)
Step 9: Monitoring
METRICS (alert thresholds):
Delivery success rate: target >99%
IF <95%: P1 alert
First-attempt success: target >95%
IF <90%: investigate endpoint health
P50 delivery latency: <500ms
P99 delivery latency: <5s
DLQ entries: alert if >100 pending
Open circuit breakers: alert if >0
Retry queue depth: alert if >10000
Step 10: Commit
Commit: "webhook: <desc> -- <components>"
Key Behaviors
- Sign everything. HMAC-SHA256 on every delivery.
- Retry with backoff + jitter. 10 retries, 3 days.
- Circuit break bad endpoints. Stop hammering.
- Dead letter with replay. No silent data loss.
- Verify on receive. Raw bytes, constant-time.
- Idempotency mandatory. Both inbound and outbound.
- HTTPS only. Block private IPs (SSRF).
- Never ask to continue. Loop autonomously.
HARD RULES
- NEVER deliver webhooks synchronously in request.
- NEVER retry on 4xx (except 408/429).
- NEVER use simple string comparison for HMAC. Constant-time only (timingSafeEqual).
- NEVER parse body before verifying signature.
- NEVER allow HTTP webhook URLs in production.
- NEVER skip SSRF checks on callback URLs.
- ALWAYS send failed events to DLQ.
- ALWAYS add jitter to retry delays.
Auto-Detection
grep -r "svix\|standardwebhooks" package.json \
2>/dev/null
grep -r "createHmac\|timingSafeEqual" \
src/ --include="*.ts" -l 2>/dev/null
grep -r "webhook" migrations/ --include="*.sql" \
-l 2>/dev/null
TSV Logging
Log to .godmode/webhook-results.tsv:
timestamp\tproject\tdirection\tevent_types\tretry\tdlq\tssrf\tcommit
Output Format
Webhook: {endpoints} endpoints, {issues} issues
-> {fixed} fixed. Status: {DONE|PARTIAL}.
Keep/Discard Discipline
KEEP if: delivery succeeds AND retry works
AND idempotency verified AND HMAC validates
DISCARD if: delivery fails after 10 retries
OR duplicate processing detected
Revert: git reset --hard HEAD~1
Stop Conditions
STOP when FIRST of:
- All endpoints verified (retry + idempotency)
- Max iterations reached (budget exhausted)
- 3 consecutive iterations <1% improvement
- >5 consecutive discards