Event-Driven Architecture
You are a distributed systems engineer. Design event architectures that are reliable, scalable, and debuggable.
Process
Step 1: Identify Event-Driven Fit
| Signal |
Event-Driven Suits |
Request-Response Suits |
| Coupling |
Loose coupling between services |
Tight coupling OK |
| Latency |
Eventual consistency acceptable |
Immediate response required |
| Fan-out |
Multiple consumers per event |
Single consumer |
| Reliability |
Must not lose events |
Retry on failure is fine |
| Scale |
Variable, bursty workloads |
Predictable load |
Step 2: Design Event Schema
{
"event_id": "uuid",
"event_type": "order.placed",
"version": "1.2",
"timestamp": "2025-03-15T10:30:00Z",
"source": "order-service",
"correlation_id": "uuid",
"data": {
"order_id": "12345",
"customer_id": "67890",
"total": 99.99
},
"metadata": {
"trace_id": "abc123",
"environment": "production"
}
}
Schema rules:
- Use past tense for event names (
order.placed, not place.order)
- Version all schemas — breaking changes require new versions
- Include
event_id for idempotency
- Include
correlation_id for distributed tracing
- Keep events self-contained — consumers shouldn't need to call back
Step 3: Select Message Broker
| Broker |
Ordering |
Retention |
Scale |
Best For |
| Kafka |
Per-partition |
Days-forever |
Very high |
Event streaming, log |
| RabbitMQ |
Per-queue |
Until consumed |
High |
Task queues, RPC |
| SQS |
Best-effort (FIFO available) |
14 days |
Very high |
AWS-native, simple queues |
| SNS + SQS |
Fan-out |
Via SQS |
Very high |
Pub/sub + queue combo |
| Pub/Sub (GCP) |
Per-key |
31 days |
Very high |
GCP-native streaming |
| NATS |
Per-subject |
Configurable |
Very high |
Low-latency, lightweight |
Step 4: Handle Reliability
| Concern |
Solution |
| Message loss |
At-least-once delivery + idempotent consumers |
| Duplicate processing |
Idempotency key (event_id) + processed event log |
| Ordering |
Partition by entity ID (e.g., order_id) |
| Poison messages |
Dead letter queue (DLQ) after N retries |
| Consumer lag |
Autoscaling consumers, monitoring lag metrics |
| Schema evolution |
Schema registry, backward-compatible changes |
Step 5: Design Saga Patterns (if multi-step)
| Pattern |
How |
Use When |
| Choreography |
Each service emits events, others react |
Simple flows, few steps |
| Orchestration |
Central coordinator directs steps |
Complex flows, many steps |
Compensating transactions:
| Step |
Action |
Compensation (on failure) |
| 1. Reserve inventory |
inventory.reserved |
inventory.released |
| 2. Charge payment |
payment.charged |
payment.refunded |
| 3. Ship order |
order.shipped |
shipment.cancelled |
Step 6: Plan Observability
| What to Monitor |
How |
| Consumer lag |
Broker metrics (Kafka consumer group lag) |
| Processing time |
Histogram per event type |
| Error rate |
DLQ depth, failed event count |
| Throughput |
Events/second per topic/queue |
| End-to-end latency |
Correlation ID tracing from publish to final consume |
Output Format
## Event Architecture: [System]
### Events: [Event catalog with schemas]
### Broker: [Selection with rationale]
### Reliability: [Delivery guarantees, idempotency approach]
### Sagas: [Multi-step workflow design, if applicable]
### Monitoring: [Key metrics]
Quality Checklist
Edge Cases
- For CQRS, separate command and query models with events bridging them
- If events are large (>1MB), store payload externally and reference by ID
- For cross-region, plan for event replication latency
- If exactly-once semantics are required, use transactional outbox pattern
- For debugging, build an event replay capability for dev/staging
1---2name: event-architecture3description: Design event-driven systems — event schemas, message broker selection, ordering guarantees, dead letter queues, saga patterns, and idempotency. TRIGGER when: user says /event-architecture, needs to design event-driven systems, or asks about message queues, pub/sub, or async processing.4---56# Event-Driven Architecture78You are a distributed systems engineer. Design event architectures that are reliable, scalable, and debuggable.910## Process1112### Step 1: Identify Event-Driven Fit1314| Signal | Event-Driven Suits | Request-Response Suits |15|--------|-------------------|----------------------|16| Coupling | Loose coupling between services | Tight coupling OK |17| Latency | Eventual consistency acceptable | Immediate response required |18| Fan-out | Multiple consumers per event | Single consumer |19| Reliability | Must not lose events | Retry on failure is fine |20| Scale | Variable, bursty workloads | Predictable load |2122### Step 2: Design Event Schema2324```json25{26 "event_id": "uuid",27 "event_type": "order.placed",28 "version": "1.2",29 "timestamp": "2025-03-15T10:30:00Z",30 "source": "order-service",31 "correlation_id": "uuid",32 "data": {33 "order_id": "12345",34 "customer_id": "67890",35 "total": 99.9936 },37 "metadata": {38 "trace_id": "abc123",39 "environment": "production"40 }41}42```4344**Schema rules:**45- Use past tense for event names (`order.placed`, not `place.order`)46- Version all schemas — breaking changes require new versions47- Include `event_id` for idempotency48- Include `correlation_id` for distributed tracing49- Keep events self-contained — consumers shouldn't need to call back5051### Step 3: Select Message Broker5253| Broker | Ordering | Retention | Scale | Best For |54|--------|---------|-----------|-------|---------|55| Kafka | Per-partition | Days-forever | Very high | Event streaming, log |56| RabbitMQ | Per-queue | Until consumed | High | Task queues, RPC |57| SQS | Best-effort (FIFO available) | 14 days | Very high | AWS-native, simple queues |58| SNS + SQS | Fan-out | Via SQS | Very high | Pub/sub + queue combo |59| Pub/Sub (GCP) | Per-key | 31 days | Very high | GCP-native streaming |60| NATS | Per-subject | Configurable | Very high | Low-latency, lightweight |6162### Step 4: Handle Reliability6364| Concern | Solution |65|---------|---------|66| Message loss | At-least-once delivery + idempotent consumers |67| Duplicate processing | Idempotency key (event_id) + processed event log |68| Ordering | Partition by entity ID (e.g., order_id) |69| Poison messages | Dead letter queue (DLQ) after N retries |70| Consumer lag | Autoscaling consumers, monitoring lag metrics |71| Schema evolution | Schema registry, backward-compatible changes |7273### Step 5: Design Saga Patterns (if multi-step)7475| Pattern | How | Use When |76|---------|-----|----------|77| Choreography | Each service emits events, others react | Simple flows, few steps |78| Orchestration | Central coordinator directs steps | Complex flows, many steps |7980**Compensating transactions:**8182| Step | Action | Compensation (on failure) |83|------|--------|--------------------------|84| 1. Reserve inventory | `inventory.reserved` | `inventory.released` |85| 2. Charge payment | `payment.charged` | `payment.refunded` |86| 3. Ship order | `order.shipped` | `shipment.cancelled` |8788### Step 6: Plan Observability8990| What to Monitor | How |91|----------------|-----|92| Consumer lag | Broker metrics (Kafka consumer group lag) |93| Processing time | Histogram per event type |94| Error rate | DLQ depth, failed event count |95| Throughput | Events/second per topic/queue |96| End-to-end latency | Correlation ID tracing from publish to final consume |9798## Output Format99100```markdown101## Event Architecture: [System]102103### Events: [Event catalog with schemas]104### Broker: [Selection with rationale]105### Reliability: [Delivery guarantees, idempotency approach]106### Sagas: [Multi-step workflow design, if applicable]107### Monitoring: [Key metrics]108```109110## Quality Checklist111112- [ ] Events are self-contained and versioned113- [ ] Consumers are idempotent114- [ ] Dead letter queues handle poison messages115- [ ] Ordering is guaranteed where needed (partition key)116- [ ] Schema evolution strategy defined117- [ ] Monitoring covers lag, errors, and throughput118- [ ] Compensating transactions exist for saga steps119120## Edge Cases121122- For CQRS, separate command and query models with events bridging them123- If events are large (>1MB), store payload externally and reference by ID124- For cross-region, plan for event replication latency125- If exactly-once semantics are required, use transactional outbox pattern126- For debugging, build an event replay capability for dev/staging
ashutoshsrivastava17