Integration Patterns
Design robust data integrations between systems using time-tested patterns and technologies.
Context
You are integrating data from multiple systems. Analyze source systems, target requirements, consistency constraints, and operational overhead. Read existing integration code or architecture documents.
Domain Context
Based on Gregor Hohpe's Enterprise Integration Patterns and modern streaming:
- ETL (Extract-Transform-Load): Scheduled batch jobs; simple but high latency
- ELT (Extract-Load-Transform): Load raw data first, transform in warehouse; flexible but storage intensive
- CDC (Change Data Capture): Capture database changes in realtime; high fidelity but complex
- Event Streaming: Publish domain events to message brokers; loosely coupled but at-least-once semantics
- API-Based Integration: Synchronous queries; simple but tightly coupled and slower
Instructions
Choose Primary Pattern: Batch ETL (cost-effective, simple)? CDC (realtime, high fidelity)? Event streams (loosely coupled, asynchronous)? Synchronous APIs (simple, tight coupling)? Often hybrid approach.
Design Idempotency: Ensure processing same data twice yields same result. Use unique identifiers, deduplication logic, or transactional sinks.
Handle Schema Evolution: Source schema changes. Build transformation layer that's resilient to new fields, deprecated fields, type changes.
Implement Error Handling: Poison pill messages, dead-letter queues, circuit breakers. Log failures with context for debugging and replay.
Plan Monitoring and Alerting: Track pipeline freshness (time since last successful run), volume anomalies, latency, error rates. Alert on SLA violations.
Anti-Patterns
- Direct Database-to-Database Integration: Source DB talks directly to sink. Result: tight coupling, hard to recover from failures, operational complexity. Guard: Introduce message queue or ETL layer for decoupling.
- No Idempotency Guarantees: Retry logic assumes exactly-once delivery. Result: duplicate data, inconsistency. Guard: Always design for at-least-once; implement idempotency at sink.
- Ignoring Backpressure: Source pushes data faster than sink can consume. Result: memory buildup, crashes. Guard: Implement queueing, rate limiting, or load shedding.
- Monolithic Transformation Logic: All business logic in single transformation. Result: hard to test, reuse, evolve. Guard: Break into modular stages; test each independently.
Further Reading
- Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf — foundational patterns
- Designing Event-Driven Systems by Ben Stopford — event-based integration
- Kafka: The Definitive Guide by Neha Narkhede — modern stream-based integration
1---2name: integration-patterns3description: Integrate data across systems using ETL, CDC, event streaming, and API patterns. Design robust, maintainable data integrations. Use when connecting disparate data sources or microservices.4---56# Integration Patterns78Design robust data integrations between systems using time-tested patterns and technologies.910## Context1112You are integrating data from multiple systems. Analyze source systems, target requirements, consistency constraints, and operational overhead. Read existing integration code or architecture documents.1314## Domain Context1516Based on Gregor Hohpe's _Enterprise Integration Patterns_ and modern streaming:1718- **ETL (Extract-Transform-Load)**: Scheduled batch jobs; simple but high latency19- **ELT (Extract-Load-Transform)**: Load raw data first, transform in warehouse; flexible but storage intensive20- **CDC (Change Data Capture)**: Capture database changes in realtime; high fidelity but complex21- **Event Streaming**: Publish domain events to message brokers; loosely coupled but at-least-once semantics22- **API-Based Integration**: Synchronous queries; simple but tightly coupled and slower2324## Instructions25261. **Choose Primary Pattern**: Batch ETL (cost-effective, simple)? CDC (realtime, high fidelity)? Event streams (loosely coupled, asynchronous)? Synchronous APIs (simple, tight coupling)? Often hybrid approach.27282. **Design Idempotency**: Ensure processing same data twice yields same result. Use unique identifiers, deduplication logic, or transactional sinks.29303. **Handle Schema Evolution**: Source schema changes. Build transformation layer that's resilient to new fields, deprecated fields, type changes.31324. **Implement Error Handling**: Poison pill messages, dead-letter queues, circuit breakers. Log failures with context for debugging and replay.33345. **Plan Monitoring and Alerting**: Track pipeline freshness (time since last successful run), volume anomalies, latency, error rates. Alert on SLA violations.3536## Anti-Patterns3738- **Direct Database-to-Database Integration**: Source DB talks directly to sink. Result: tight coupling, hard to recover from failures, operational complexity. **Guard**: Introduce message queue or ETL layer for decoupling.39- **No Idempotency Guarantees**: Retry logic assumes exactly-once delivery. Result: duplicate data, inconsistency. **Guard**: Always design for at-least-once; implement idempotency at sink.40- **Ignoring Backpressure**: Source pushes data faster than sink can consume. Result: memory buildup, crashes. **Guard**: Implement queueing, rate limiting, or load shedding.41- **Monolithic Transformation Logic**: All business logic in single transformation. Result: hard to test, reuse, evolve. **Guard**: Break into modular stages; test each independently.4243## Further Reading4445- _Enterprise Integration Patterns_ by Gregor Hohpe and Bobby Woolf — foundational patterns46- _Designing Event-Driven Systems_ by Ben Stopford — event-based integration47- _Kafka: The Definitive Guide_ by Neha Narkhede — modern stream-based integration
sethdford