vaquarkhan Debezium And Kafka Connect CDC
Overview
Use this skill when database changes must be captured and delivered through Debezium and Kafka Connect. It helps agents define connector safety, snapshot behavior, schema handling, and downstream CDC contracts.
When to Use
- setting up
Debeziumconnectors forPostgreSQL,MySQL,MongoDB,SQL Server, orOracle - designing Kafka-based CDC from transactional databases
- handling initial snapshots, incremental streaming, and connector recovery
- feeding downstream stream processors, lakehouse sinks, or search indexes
- managing schema evolution when source tables change
Do not use this when changes can be captured through application-level events or batch extracts are sufficient for freshness requirements.
Workflow
Define source tables, keys, and change event semantics. Include:
- which tables to capture and which to exclude
- primary key and unique key strategy for each table
- expected change operations: inserts, updates, deletes, truncates
- whether tombstone records are needed for compacted topics
- expected change volume and peak throughput
Plan snapshot behavior explicitly.
- initial snapshot: full table scan on first connector start
- define snapshot mode:
initial,schema_only,never, orwhen_needed - understand lock behavior during snapshot (especially for
PostgreSQLandMySQL) - plan for snapshot duration on large tables — can take hours
- document what happens if a snapshot is interrupted
Configure connector for operational resilience.
- set appropriate
max.batch.sizeandpoll.interval.ms - configure heartbeat intervals to prevent WAL/binlog retention issues
- define slot or binlog retention policies on the source database
- plan for connector task failures and automatic restarts
- monitor connector lag and offset position
- set appropriate
Define topic contracts and downstream expectations.
- topic naming convention:
{prefix}.{schema}.{table} - event envelope format: include before/after, operation type, source metadata
- schema registry integration for event contracts
- retention and compaction policy per topic
- document which consumers depend on each topic
- topic naming convention:
Handle schema evolution safely.
- source DDL changes (column adds, renames, type changes) propagate through CDC
- define compatibility policy in schema registry (backward, forward, full)
- test that downstream consumers handle schema changes without failure
- plan for breaking changes: connector restart, re-snapshot, or topic migration
Plan recovery, replay, and operational safety.
- define what happens when the connector falls too far behind (slot overflow, binlog expiry)
- document re-snapshot procedure for recovery
- plan for database failover: does the connector reconnect automatically?
- define monitoring alerts: lag, errors, rebalances, and dead-letter routing
Common Rationalizations
| Rationalization | Reality |
|---|---|
| "CDC is just a connector — set it and forget it." | Connectors require active monitoring, offset management, and schema evolution handling. Neglected connectors silently lose data. |
| "We don't need to worry about snapshots after the first one." | Database failovers, slot loss, and connector resets can trigger re-snapshots. The team must plan for snapshot impact on source load. |
| "Schema changes in the source don't affect CDC." | Every DDL change propagates through the CDC stream. Without compatibility policies, downstream consumers break silently. |
| "Kafka handles deduplication for us." | Debezium provides at-least-once delivery. Consumers must handle duplicates or use idempotent processing patterns. |
Red Flags
- no monitoring of connector lag or WAL/binlog retention
- snapshot mode is undefined or set to
alwayswithout understanding impact - no schema registry integration for CDC topics
- downstream consumers assume exactly-once delivery without deduplication logic
- heartbeat intervals are not configured, risking WAL bloat on the source
- no documented recovery procedure for slot loss or binlog expiry
- topic retention is unlimited with no compaction policy
- connector runs with a single task on high-volume multi-table sources
Verification
- Source tables, keys, and change semantics are explicitly documented
- Snapshot behavior and impact on source database are understood and planned
- Topic contracts include naming, envelope format, retention, and schema compatibility
- Schema evolution paths are tested for both additive and breaking changes
- Connector monitoring covers lag, errors, rebalances, and offset position
- Recovery procedures for slot loss, binlog expiry, and database failover are documented
- Downstream consumers handle at-least-once delivery with deduplication or idempotency