Streaming and change data capture (CDC) standards
Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).
Hard premise: "real time" is, alongside "lakehouse", the most oversold and worst justified
label in the industry. An incremental batch every 15 minutes solves the vast majority of
"real-time requirements", and streaming is not an execution mode: it is a permanent
operation that somebody has to watch every night of the year. Start by proving it is needed.
1. Scope and triggers
Applies to capturing changes from a source system and to processing them in motion: the
decision of whether streaming is needed at all, the capture mechanism, operating the capturer and
its impact on the source, delivery semantics, ordering, schema evolution of the stream, stateful and
time-based processing, and the operation and cost of all of it.
Triggers: debezium, debezium-server, DebeziumEngine, io.debezium.connector.*,
snapshot.mode, signal.data.collection, tombstone, transforms=unwrap,
ExtractNewRecordState, pgoutput, wal2json, CREATE PUBLICATION,
pg_create_logical_replication_slot, pg_replication_slots, slot_name, restart_lsn,
wal_status, safe_wal_size, max_slot_wal_keep_size, idle_replication_slot_timeout,
REPLICA IDENTITY FULL, binlog, gtid, server-id, LogMiner, change streams,
kafka-connect, connect-distributed.properties, consumer lag, auto.offset.reset,
enable.idempotence, isolation.level=read_committed, transactional.id, StreamsBuilder,
KTable/KStream, readStream/writeStream, withWatermark, checkpointLocation,
StreamExecutionEnvironment, KeyedProcessFunction, WatermarkStrategy, savepoint,
checkpoint, DLQ/dead letter, and the phrases of the domain: "we want it in real time", "the
primary's disk filled up", "the connector has been down since Friday", "the deletes are missing",
"duplicate events are arriving", "events arrive out of order", "we have to reprocess the whole
history", "the source added a column and the consumer broke".
Not applicable: see
data-platform-standards (mother): Kafka as an engine is hers —topic partitions,
retention, replicas, min.insync.replicas, KRaft— and so is the schema registry (Karapace,
Apicurio, Confluent Schema Registry) and PostgreSQL as an engine with its replication, HA and
PITR. Here, what gets published on those topics, with what guarantees and what is done with the
stream. Her principle —one store because it is needed, not because it is fashionable— is
inherited as "one stream because it is needed, not because it is fashionable" (§2.1).
message-brokers-standards — this is the most likely collision of the pair and
the line is written here with precision: the broker as a component is hers (Kafka, Pulsar,
RabbitMQ, NATS: choice, topology, sizing, replication, retention, cluster operation, queues versus
logs). Processing the stream and capturing what goes into it belong here. Mechanical cut-off
rule: if the broker makes the decision, it is hers; if the producer, the consumer or the
processor makes it, it is ours. "How many partitions can the cluster take?" is hers; "which
partition key do I choose and what ordering does it guarantee me?" is ours.
microservices-architecture-standards (critical boundary; see §3.3, it is the most expensive
design mistake in the domain): the outbox pattern, sagas, domain events and per-service data
ownership are hers. A domain event is designed and published by the application that owns
the data; a row change captured by CDC is not a domain event. Here, the capture and transport
mechanism — including CDC over the outbox table, which is its correct use.
lakehouse-standards (sister; boundary declared on both sides): what happens to the table
when the changes land is hers —copy-on-write versus merge-on-read, cost of
MERGE/DELETE, files and deletion vectors, compaction, snapshots, catalog—. Here, how the
changes are captured and how they are processed on the way there. CDC is a typical lakehouse
source, not the only one (the lakehouse is fed by batch just the same) and the lakehouse is
not the only CDC destination (a replica, a cache, a search index, another service). Neither
presupposes the other.
data-engineering-standards (sister): batch ingestion, orchestration, the watermark of an
incremental, job idempotency, backfill and file format. She sets when change capture is the
answer; here how it is done well. Her mirrored rule: the file format is hers, the
table format belongs to lakehouse-standards.
data-governance-quality-standards (already on disk): data contracts, ownership, catalog,
lineage and quality as a programme are hers. Here, their execution in the stream: which schema
gets published, who the responsible producer is and which assertions divert an event to the failed
queue. Cut-off rule: if the question is "who is the owner and what does the contract promise?", it
is hers; if it is "what does the consumer do when an event arrives that breaks it?", it is ours.
data-warehouse-modeling-standards: the destination model. A change stream is not an SCD2:
dimensional historisation is a modelling decision of hers; here the change is merely delivered.
privacy-engineering-standards: personal data, minimisation, retention and erasure are hers.
Here their technical constraint: a change published on a log with long retention is one more
copy of the personal data, and a DELETE at the source does not erase the previous events in
the log (§5).
observability-standards: instrumentation, OTel, cardinality and the metrics backend. Here,
what to measure in a stream (§6).
sre-practice-standards (SLO, error budget, on-call), incident-management-standards,
backup-recovery-standards (the event log is not a backup),
bcdr-standards, kubernetes-standards, iac-standards, cicd-standards,
secrets-management-standards, identity-access-management-standards,
grc-compliance-standards, mlops-standards, nosql-standards, search-engines-standards,
vector-db-standards, graph-db-standards, timeseries-db-standards,
oracle-dba-standards/sqlserver-dba-standards/mysql-mariadb-dba-standards (tuning and
operating the source engine are theirs; here only what CDC demands of it. Arbitration rule
mirrored with mysql-mariadb-dba-standards: capture is ours, impact on the engine is theirs
— the GTID format, the binlog as an engine mechanism and its purging are theirs; reading it as
a change stream is ours), ibm-i-rpg-standards (the Db2 for i equivalent, with the same
mirrored rule: the journal and its receivers —CRTJRN, STRJRNPF, IMAGES(*BOTH),
receiver management and purging— are hers, along with the platform's data traps (packed decimal,
numeric dates, multi-member files, CCSID 65535); reading that journal as a change stream is
ours. It is declared explicitly because the mechanism catalogue in §2.2 did not include it,
and it must not be inferred from that that the platform has no log-based capture: it has one, and
it is among the oldest), aws-standards/azure-standards/gcp-standards (MSK, DMS, Kinesis, Event Hubs,
Pub/Sub, Datastream as managed services), python-standards, jvm-spring-standards
(Java/Kotlin code quality: Flink and Kafka Streams are JVM and their build, tests and
packaging are hers; stream design is ours), scala-standards (same boundary for
the Scala API of Flink and Kafka Streams), sql-standards (Flink SQL and ksqlDB have
streaming semantics of their own —windows, watermarks, dynamic tables— which are ours; the
relational SQL written against a sink is hers).
Governing principle: streaming is not paid for in compute, it is paid for in continuous
operation. A batch that fails is retried tomorrow; a stream that fails accumulates lag, holds
resources at the source and degrades while nobody is looking. Before adopting it, the question is
not "can I?", it is "who watches it at 3 a.m. on a Sunday in August?".
2. Default decisions
Verify version, licence and ownership on the web before committing to it in a real project (§8).
In 2026 this subsector changed hands: IBM completed the acquisition of Confluent on
17 Mar 2026.
2.1 The starting decision: do you really need streaming?
Exhaust this ladder before introducing a stream. Every rung avoided is a permanent operation you
do not take on:
| Stated need |
Simplest solution |
When it genuinely stops working |
| "We want it in real time" (for a report that is looked at in the morning) |
Incremental batch by watermark (see data-engineering-standards) |
Never stops working. This case is not streaming |
| Latency of minutes, moderate volume |
Incremental batch every 5-15 minutes |
When the source has no reliable watermark, or deletes must be captured |
| Query operational data with minimal delay |
Read replica of the engine itself (see data-platform-standards) |
When the destination is another engine, or another data model |
| You need deletes, ordering and the complete history of changes |
CDC from the transaction log |
— |
| Somebody or something acts within seconds on the event (fraud, stock, alerts, prices) |
Streaming with stateful processing |
— |
Two questions that settle most arguments:
- Which decision changes if the data arrives in 5 seconds instead of 15 minutes, and who makes
it? If the answer is "none" or "nobody", the real-time requirement does not exist.
- What is the cost of the stream being down for 4 hours on a Sunday? If it is "none", you do
not need streaming either. If it is "severe", you need on-call, not just technology.
Honest rule: CDC is the right answer to a problem other than real time. Its real value is
usually capturing deletes and rapid changes faithfully, not latency. Many projects that "need
streaming" actually needed CDC dumping into a destination in micro-batches.
2.2 How changes are captured: the mechanism matters more than the tool
| Mechanism |
How it works |
Verdict |
Reading the transaction log (PostgreSQL WAL, MySQL/MariaDB binlog, Oracle LogMiner, SQL Server CDC, MongoDB change streams, Db2 for i journals and receivers — see ibm-i-rpg-standards) |
Reads the engine's write log: it sees every operation, in commit order, with before and after |
The correct way. No schema impact, captures deletes, respects transactional ordering. Absolute default |
| Triggers writing to an audit table |
The engine runs code on every write |
Vetoed unless the log is impossible. Penalises every transaction of the operational system, is bypassed by badly written bulk operations, pollutes the schema and is forgotten when a new table is created |
Timestamp query (WHERE updated_at > :x) |
Periodic polling |
This is not CDC. It loses physical deletes (for ever and silently) and loses multiple changes between polls (you only see the last state). Besides, updated_at is almost never reliable: skewed clocks, bulk updates that do not touch it, writes that commit out of order. Legitimate only as an incremental batch, knowing what is being lost |
| Dual write from the application (DB and then broker) |
The app writes to both places |
FORBIDDEN (§7): it is not atomic. See outbox in microservices-architecture-standards |
A warning about watermarks and commit ordering: even with a perfect updated_at, a long
transaction may commit after your window has already passed its timestamp. Without window
overlap, that row is never ingested. It is the classic silent loss and it is one of the strongest
arguments in favour of the log.
2.3 Toolchain
| Area |
Default |
Verified status (Aug 2026) |
Justifiable alternative |
| CDC capturer |
Debezium 3.6.x |
3.6.0.Final (1 Jul 2026); 3.7.0.Alpha1 in progress. Apache 2.0 (verified in the repository). De facto standard, backed by Red Hat |
Flink CDC (Apache 2.0) if processing is already Flink and you want a single system |
| Running the capturer |
Debezium Server (standalone) if you do not need Kafka Connect |
Exists and is maintained; in the 3.6 series it is migrating from the old embedded engine to the Quarkus extension (debezium-quarkus-engine), with lifecycle control for hot standby with external locking. Careful: it is an architectural migration in progress, not a consolidated mode |
Distributed Kafka Connect when you already operate Connect: it remains the most proven route and the one that has had the most eyes on it |
| Broker |
Kafka (the component and its operation belong to data-platform-standards / message-brokers-standards) |
Kafka 4.3.1 (Jun 2026), Apache 2.0, ASF. IBM's acquisition of Confluent does not change Kafka's licence: the project belongs to the ASF and its PMC is independent |
— |
| Demanding stateful processing |
Apache Flink 2.3.0 |
2.3.0 stable according to the official downloads page (Jun 2026); 2.2.1, 2.1.3 and 1.20.5 also published. It does not lose the technical crown, but it does lose new teams because of its operational complexity: it demands dedicated people |
Managed Flink service if you have no platform |
| Processing embedded in a JVM service |
Kafka Streams |
Alive. No cluster of its own; bounded by the number of partitions and with less sophisticated time/late-data handling than Flink |
— |
| Processing where Spark already exists |
Spark Structured Streaming |
Alive (Spark 4.2.0, Jul 2026). Micro-batches: latency of seconds, not milliseconds. Wins by inertia if the team is already Spark |
— |
| Managed SQL-first engines (RisingWave, Materialize, Decodable, ksqlDB…) |
— |
They greatly reduce the operational load in exchange for flexibility and for lock-in |
Valid with an ADR and an exit plan |
| Event format |
Avro or Protobuf with a schema registry |
The registry belongs to data-platform-standards |
JSON only with validation in CI |
Criterion for choosing the processor, one line each:
- Flink if there is large state, serious event-time semantics and late data that matters,
and you have somebody to operate it.
- Kafka Streams if it is a JVM application that already lives inside Kafka and you do not
want another cluster.
- Spark Structured Streaming if you already operate Spark and second-level latency is
enough.
- None of them if what you needed was an incremental batch (§2.1). It is the right answer more
often than is admitted.
Retired or declining engines — do not choose them for anything new: classic Spark Streaming
(DStreams) is obsolete compared with Structured Streaming; Apache Apex has had no development since
2019; Samza has not released since 2023; Storm is maintained but is rarely the right choice today.
Ownership and licence risk: IBM completed the purchase of Confluent on 17 Mar 2026 (around
$11bn; WarpStream included). Kafka as a project is unaffected, but the Confluent Community
License —which covers Schema Registry, REST Proxy, ksqlDB and several connectors— is not OSI
open source and forbids offering them as a competing service. With the new owner, the pricing and
support model is a risk to watch, not a settled fact (§8). Practical consequence: prefer Karapace
or Apicurio for the registry (criterion already set in data-platform-standards) and record in
the ADR which non-OSI-licensed components you depend on.
3. Structure and conventions
3.1 CDC in practice: the initial snapshot and resuming
- The initial snapshot is the most expensive and most dangerous moment of the whole project:
reading an entire table on a production system. Rules:
- Read from a replica whenever the mechanism allows it; if not, plan it in a low-load window,
with bounded reads and
statement_timeout.
- Prefer the incremental snapshot (chunked, resumable, in parallel with the stream) over the
blocking one. A snapshot that cannot be resumed turns into five twelve-hour retries.
- Measure and budget how long it takes before running it in production. It is the figure that
decides whether the project is viable.
- Re-snapshotting has to be a planned and rehearsed operation (signals / ad hoc snapshot),
not an accident: you will need it when a slot is invalidated or a position is lost.
- Resuming after a crash: the capturer stores its offset (LSN, GTID, resume token)
durably and outside its own process. Rules: the offset store goes into the backup; if it is
lost, the price is a full new snapshot; and the offset is committed after delivering, never
before — the other way round loses changes silently, this way they merely duplicate (and that
is what idempotency is for, §3.4).
- Heartbeats: in databases with many tables where you only capture a few, and in databases with
little activity, the capturer may not advance its offset for hours while the WAL piles up.
Enable heartbeats; it is the specific mitigation and almost nobody knows about it.
- Filter at the source, not at the destination: include only the necessary tables and columns
(bounded publications, include lists). Less load, less personal-data surface (§5), less schema
noise.
3.2 The impact on the source: the production failure par excellence
A logical replication slot retains WAL for as long as its consumer does not confirm. If the
consumer stops and nobody notices, pg_wal grows until the disk fills and the primary goes down.
This is not a rare case: it is the incident of this domain. Verified against PostgreSQL 18:
max_slot_wal_keep_size (since PG 13; reloadable via SIGHUP, in MB if no unit is given)
limits the WAL that slots may retain. With the default value -1 retention is unlimited,
which is exactly the configuration that fills the disk. Always set it to a value consistent with
the real free space.
- The trade-off has to be accepted explicitly: when the limit is exceeded, PostgreSQL
invalidates the slot and recycles the WAL. The consumer can no longer continue and a new
slot and, typically, a full initial snapshot will be needed. It is a conscious decision: you
sacrifice CDC to save the database. It is the right choice, and it has to be written in the
runbook before it happens.
idle_replication_slot_timeout (new in PostgreSQL 18; unitless value in seconds, 0
disables it) automatically invalidates inactive slots. Invalidation happens at the
checkpoint, so there is a delay between crossing the threshold and the actual invalidation. It
does not apply to slots that do not reserve WAL nor to slots synchronised from the primary
(synced = true). Reasonable values are of the order of 48-72 h.
- Mandatory monitoring of
pg_replication_slots: active, wal_status, safe_wal_size
(with a positive max_slot_wal_keep_size it indicates how much WAL is left before invalidation;
negative means it has already been passed) and invalidation_reason. Alert with a threshold
on slot WAL retention and on inactive slots, without exception. And delete orphan slots: a
slot from a test that nobody removed is a time bomb.
- Long transactions at the source make everything worse: they retain WAL and can fill the
storage on their own. Complement with
wal_sender_timeout (60 s by default) to detect dead
consumers.
REPLICA IDENTITY: by default PostgreSQL only publishes the primary key in the "before" of an
UPDATE/DELETE. REPLICA IDENTITY FULL gives the full row at the cost of more WAL and more
load: enable it per table and only if the consumer needs it, never globally.
- Other engines, same principle: in MySQL/MariaDB, binlog retention (
binlog_expire_logs_seconds)
sets your resume window — if the consumer is down for longer than that retention, the position
is lost and a re-snapshot is required. In Oracle, log mining has a cost in the engine itself.
Coordinate the parameter with the engine's DBA, it is a shared decision.
- Cross-cutting rule: CDC couples the health of your analytics platform to the health of the
operational database. It is the hidden cost of the technique and it has to be told in writing to
the team that owns the source before anything is switched on.
3.3 CDC versus outbox — the boundary and the most expensive design mistake in the domain
Say it plainly and without lukewarm qualifiers: table changes captured by CDC are NOT domain
events. An UPDATE on orders tells you that some columns changed value; it does not tell you
that the order was cancelled, nor why, nor by whom. Publishing raw CDC as a contract between
services means:
- You couple every consumer to the internal physical schema of the source service. From that
moment on, that service can no longer refactor its database: any
ALTER TABLE is a breaking change
in a public API that nobody declared as such. It is exactly the same mistake as letting another
service query your database, with an intermediate hop that disguises it.
- You lose intent. The row diff does not contain the "why". Each consumer reconstructs the
semantics its own way, and all the reconstructions are different and plausible.
- You expose everything that is in the table, including columns that should never have left the
service (§5).
- No contract and no versioning: a refactor breaks consumers in production without prior
warning.
The correct division, and the boundary with microservices-architecture-standards:
| Case |
Correct pattern |
Owner of the decision |
| Publish that something of business significance happened so other services react |
Outbox: the application writes the domain event, with its schema and its version, in the same transaction as the state change |
microservices-architecture-standards |
| Transport that event reliably from the outbox table to the broker |
CDC over the outbox table — this is the correct and virtuous use of CDC in service architecture |
Here |
| Feed an analytical store, a lakehouse, a cache or a search index from existing tables |
Direct CDC over the tables, with the destination treated as an internal consumer |
Here |
| Integrate a legacy system that cannot be modified |
Direct CDC, with a mandatory anti-corruption layer translating to a contract of your own before exposing it to anybody else |
Here |
Hard rules that follow:
- A raw CDC topic is never a public contract. If somebody outside the owning team consumes it,
either it becomes a domain event with an outbox, or an anti-corruption layer with its own versioned
schema is placed in between. No exceptions.
- The cost of the outbox is real and must be stated: it touches every write path, the table grows
and has to be purged, and you maintain two schemas (the table's and the event's). It is paid
because the contract is worth it.
- Tombstones: on a compacted topic, a delete is represented by a message with a null value. If
your consumer does not handle them, deletes are not propagated and your copy accumulates
ghosts. It is a silent failure, which is the worst kind.
3.4 Delivery semantics: what they really mean
| Semantics |
What it guarantees |
How it is achieved |
Verdict |
| At-most-once |
Never duplicates; may lose |
Commit the offset before processing |
It is almost always an accident, not a design. And it happens through carelessness |
| At-least-once |
Never loses; may duplicate |
Commit the offset after processing |
The correct default. Everything else is built on top |
| Exactly-once |
Each effect is applied once |
Transactions and state coordinated inside the system |
Real, but far narrower than it is sold as (see below) |
What "exactly once" really means: it is exactly-once of effect, not of delivery, and
only within the boundary of the system that implements it — reading from Kafka, updating state
and writing to Kafka in one transaction (Kafka Streams / transactional producer with
read_committed), or Flink's checkpoint mechanism with two-phase transactional sinks. As soon
as an effect leaves that boundary —calling an API, sending an email, writing to a database that
does not take part in the transaction— the guarantee is over. It also costs latency and
operational complexity.
Therefore: consumer idempotency is the only real defence, and it is mandatory. It is not an
alternative to exactly-once, it is its prerequisite and its substitute in 90 % of cases. It is
implemented with:
- Business key + upsert (
INSERT ... ON CONFLICT, MERGE) instead of a plain INSERT.
- Deduplication by event identifier with a bounded window, when the upsert does not apply.
- Version / sequence number comparison: discard the event whose
lsn/sequence is lower than
or equal to the one already applied. This also protects against reordering.
- Non-idempotent external effects (charging, sending, notifying) protected with an idempotency
key and an execution log. A reprocess must not bill twice.
3.5 Ordering and partitioning
- Ordering is only guaranteed within a partition. There is no global ordering in a partitioned
log, and whoever assumes there is builds a system that fails intermittently and randomly.
- Choosing the partition key is choosing what is ordered. Rule: the key is the entity whose
sequence of changes matters — typically the primary key of the source row. That way all the
changes of the same order go to the same partition and arrive in order.
- Consequences that have to be accepted:
- Partition skew: a very active entity saturates its partition and limits parallelism. It is
detected by measuring lag per partition, not the aggregate.
- Changing the number of partitions breaks key affinity and therefore historical ordering.
It is a design decision with consequences, not a capacity adjustment (topic operation belongs to
data-platform-standards).
- Consumer parallelism is bounded by the number of partitions.
- If ordering between different entities matters (a transfer between two accounts), partitioning
by entity does not give it to you. Either you put both in the same partition via a composite
business key, or you solve it with state and event time in the processor, or you redesign the
operation as a single event. Choose explicitly: do not leave it to chance.
- Reordering across topics: two different streams have no relative ordering. If a consumer needs
"customer first, then order", that is a dependency to be resolved in the consumer (wait, look
up, retry), not a transport guarantee.
3.6 Schema and evolution
The schema registry and its compatibility configuration belong to data-platform-standards. What is
specific to the stream:
- Backward compatibility (a new consumer reads old data) as the mandatory minimum;
forward (an old consumer reads new data) when you do not control the consumers, which is the
usual case. Verified in CI, not in somebody's head.
- What to do when the source adds a column: it is additive and compatible if the field has a
default value. Decide explicitly whether it is propagated (not everything that appears at the
source should leave the source, §5). If your connector has a column include list, the new column
does not get in by itself: that is the safe option.
- What to do when the source drops or renames a column: it is incompatible and breaks
consumers. It is handled as an expand/contract migration, coordinated with the team that owns
the source: publish both fields, migrate consumers, retire the old one. That the capturer
tolerates the change does not mean your consumers do.
- Type change: almost always incompatible. New field + coexistence + retirement.
- Events already published cannot be rewritten. A consumer reprocessing from the beginning of
the log will see every historical schema: your consuming code must be able to read them all
for as long as the log retains them. This is not theoretical, it is what happens on the day you
reprocess.
- The owner of the event schema is the producer, with a published and versioned contract. A
schema change without notice is an incident, not a deployment.
3.7 Processing in motion: time, windows and state
- Event time versus processing time: event time (when it happened) is the correct one for
anything the business is going to read; processing time (when the engine saw it) produces
non-reproducible results: reprocessing gives a different result from the original. Rule:
always aggregate by event time; use processing time only for telemetry of the system itself.
- Watermarks: the engine's assertion that "I no longer expect events earlier than T". It is
a heuristic, not a fact, and it embodies the central trade-off of the domain: aggressive
watermark = fast results and more data discarded; permissive watermark = late results and more
state retained. It is chosen with real lateness data from the stream, documented and reviewed.
- Data arriving late: decide and write down what happens to it. Three legitimate options —
discard it (counting it, always), accept it with an allowed lateness and re-emit the corrected
result, or divert it to a side stream for separate handling. The illegal option is not
deciding, because the silent default is to discard it without counting it, and then the figures
do not add up and nobody knows why.
- Windows: tumbling by default; sliding multiply state and compute by the overlap; session
windows when the business talks about sessions. Every open window is retained state: a 30-day
window over a high-cardinality key is a memory problem, not a requirement.
- State: it is the most expensive and most fragile asset of a stream. Rules:
- TTL on all keyed state, without exception. Keyed state with no expiry grows until it brings
the job down, and it does so months after nobody remembers the design.
- Checkpoints/savepoints on durable storage, backed up and with a tested restore. A job
whose state cannot be restored is not recoverable: only reprocessable.
- State compatibility limits deployment: changing the logic or the state schema may prevent
restoring from the previous savepoint. Decide in advance whether a deployment keeps state or
starts from scratch — and if it starts from scratch, how long it takes to catch up.
- Stateful joins over streams require retaining both sides: define the window or state
explodes.
4. Quality and testing — gates
In increasing order of cost. Those marked as gates break the build or the deployment.
- Capturer and job configuration as code, versioned and reviewed: connectors, table and column
include lists, snapshot mode, partition keys, retention, window parameters. Nothing created by
hand against a production API. CI gate.
- No secrets in the connector configuration: credentials come from the secrets manager
(see
secrets-management-standards). A connector file with the production database password is a
finding, not an oversight. CI gate.
- Schema compatibility verified in CI against the registry, with the declared policy
(minimum
BACKWARD). An incompatible change breaks the build. Gate.
- Unit tests of the processing logic with fixed events and an expected result, covering the
happy path and the mandatory edges: duplicate event, out-of-order event,
delete / tombstone, null field, late event beyond the watermark,
empty window, brand-new key never seen before, restart mid-window. CI gate.
- End-to-end idempotency test: delivering the same batch of events twice produces the same
state at the destination (same count, same checksum). Without this test, idempotency is an
intention. CI gate.
- Delete propagation test: delete at the source and check that the destination reflects the
delete. It is the most expensive silent failure and the least tested. Gate.
- Resume test: kill the capturer or the job, start it and check that it neither loses nor
duplicates in a non-idempotent way. With containers of the real engine (not with doubles),
because the log's behaviour is what is being tested.
- State restore test from checkpoint/savepoint, and of changing the application version
while keeping state. It is done before the first production deployment, not after the first
incident.
- Periodic reconciliation against the source (count and sum of a key measure, with a declared
tolerance), scheduled. A CDC stream diverges from the source over time; without reconciliation
you do not find out. This is the most important operational gate of them all.
- Reprocessing rehearsal: reprocess from the beginning of the log in a mirror environment,
measuring how long it takes and what it costs. If it does not fit in the time available, your
recovery plan does not exist.
- Dependencies pinned by hash/digest in connectors, images and job libraries (§5).
CI gate.
5. Stack security
- CDC copies everything that is in the table, including the columns that should never have left.
Filter at the capturer (column list, bounded publication, masking in the connector), not
downstream: every PII column you do not capture is a retention, erasure and breach problem you
will not have. Coordinate with
privacy-engineering-standards.
- An event log with long retention is one more copy of the personal data, with its own
classification, its declared retention and its access control. Infinite retention "just in case"
is forbidden (§7).
- The right to erasure clashes with the log: a
DELETE at the source generates a new event,
it does not erase the previous events. A compacted topic with a tombstone removes the
previous versions of that key over time, not immediately, and only if it is compacted. If
there is an erasure commitment, topic retention becomes a compliance parameter, or it is
designed with pseudonymisation / crypto-shredding from the start. The policy belongs to
privacy-engineering-standards; the technical constraint and the parameter are set here.
- Least privilege at the source: the capturer's user has only the replication/read
permissions it needs, on the agreed tables. It is not a superuser, even if the connector
tutorial says so.
- The capturer authenticates with rotatable, short-lived credentials; no eternal static
passwords in a connector file. TLS mandatory against the source and against the broker;
authorisation per topic and per consumer group.
- PII out of logs and error messages: a connector that logs the row it could not serialise is a
leak. Log key and offset, not content. The same applies to the dead-letter queue: it is a data
store with real content, and it needs its own access control and its own retention.
- Supply chain — verified precedents, not hypotheses: Trivy (Mar 2026), LiteLLM
on PyPI (1.82.7/1.82.8, 24 Mar 2026), Telnyx (Mar 2026), Microsoft's
durabletask
(1.4.1-1.4.3, May 2026) and elementary-data 0.23.3 (Apr 2026, with the .pth pattern that
runs when the interpreter starts). The current generation, "Mini Shai-Hulud"
(CVE-2026-45321), active since late Apr 2026, propagates between npm and PyPI, extracts OIDC
tokens from the memory of the GitHub Actions runner and has gone as far as forging SLSA
level 3 provenance attestations. Mandatory consequence: pin by hash/digest everything that runs
in a process holding credentials for the source database or the broker; a quarantine window of
days before adopting newly published versions; provenance attestation is no longer sufficient
proof on its own; and a runner does not hold credentials for more than one environment.
6. Performance and operability
6.1 Consumer lag is the primary metric
- Measure lag in time, not just in number of messages. "12,000 messages behind" says nothing;
"18 minutes behind" does. And measure it per partition: the aggregate hides skew,
which is the most frequent cause.
- Alert on trend, not on an isolated threshold: lag that grows steadily is an incident even if
its absolute value is still small. The fixed threshold warns when there is no margin left.
- Mandatory complementary signals: capturer lag relative to the source (which is different from
consumer lag), WAL retention / slot size (§3.2), job restart rate, retained state,
checkpoint duration and failures, message rate to the DLQ,
and data age at the destination (freshness, which is the signal the business cares about;
see
data-engineering-standards §6).
- Actionable alert with a runbook, always: what broke, what depends on it, how it is recovered,
how long it takes. A lag alert with no runbook at 4 a.m. is useless.
- If there is a latency commitment, there is on-call. An SLO of seconds with nobody to respond
out of hours is a documented lie (criterion inherited from
data-engineering-standards §6.3 and
sre-practice-standards).
6.2 Reprocessing, DLQ and the question that decides the architecture
- The dead-letter queue (DLQ) is mandatory and has an owner. Rules: every diverted message
carries the cause and the original offset; there is a written reinjection procedure; and
the DLQ is monitored with an alert. A DLQ nobody looks at is a dump of lost data with the
appearance of a solution. A poison message must never be retried infinitely: it blocks the
whole partition and stops the stream.
- Reprocessing from the beginning: it is the capability that makes a stream operable —
fixing a logic error, populating a new destination, recovering from corruption. Requirements:
idempotent consumers (§3.4), a destination that tolerates rewriting, capacity to absorb the
burst without bringing anything downstream down, and a prior measurement of how long it takes.
Rehearse it (§4).
- The question that decides the architecture: how long do you keep the log? It is not an
operational parameter, it is a structural decision:
| Retention |
What it buys you |
What it costs you |
| Short (hours/days) |
Cheap storage, small personal-data surface |
You can only reprocess the recent past. A new destination has to be seeded from the source with a snapshot |
| Medium (weeks) |
Realistic reprocessing of incidents |
Cost and data classification |
| Long or infinite (log as the source of truth) |
Rebuild any destination at any time |
Growing cost, direct conflict with retention and erasure of personal data, and every historical schema alive for ever in your consuming code |
Decide it in writing in the ADR, with the privacy team present, and be clear that the log
is not a backup: it has neither the recovery model, nor the verification, nor the isolation of
one (see backup-recovery-standards).
6.3 Cos
…(truncated)
1---2name: streaming-cdc-standards3description: Use when changes must be captured from a database or processed in motion — deciding whether streaming is warranted at all versus an incremental batch every 15 minutes, capturing from the transaction log (PostgreSQL pgoutput/wal2json logical decoding and replication slots, MySQL/MariaDB binlog and GTID, Oracle LogMiner, SQL Server CDC, MongoDB change streams) versus triggers or timestamp polling, running Debezium (Debezium Server, Debezium Engine, Kafka Connect connectors, signals, tombstone records, snapshot.mode and incremental snapshots) or Flink CDC, initial snapshot cost and resume after a crash, an inactive slot retaining WAL until pg_wal fills the disk (max_slot_wal_keep_size, idle_replication_slot_timeout, safe_wal_size, wal_status), why table-change events are not domain events and must not become a public contract, at-least-once versus at-most-once versus exactly-once and consumer idempotency, ordering guaranteed only per partition and choosing the partition key, schema evolution when the source adds4---56# Streaming and change data capture (CDC) standards78Criteria verified as of **August 2026**. Re-verify on the web before committing to anything (§8).910> **Hard premise**: "real time" is, alongside "lakehouse", the most oversold and worst justified11> label in the industry. **An incremental batch every 15 minutes solves the vast majority of12> "real-time requirements"**, and streaming is not an execution mode: it is a **permanent13> operation** that somebody has to watch every night of the year. Start by proving it is needed.1415## 1. Scope and triggers1617Applies to **capturing changes from a source system** and to **processing them in motion**: the18decision of whether streaming is needed at all, the capture mechanism, operating the capturer and19its impact on the source, delivery semantics, ordering, schema evolution of the stream, stateful and20time-based processing, and the operation and cost of all of it.2122Triggers: `debezium`, `debezium-server`, `DebeziumEngine`, `io.debezium.connector.*`,23`snapshot.mode`, `signal.data.collection`, `tombstone`, `transforms=unwrap`,24`ExtractNewRecordState`, `pgoutput`, `wal2json`, `CREATE PUBLICATION`,25`pg_create_logical_replication_slot`, `pg_replication_slots`, `slot_name`, `restart_lsn`,26`wal_status`, `safe_wal_size`, `max_slot_wal_keep_size`, `idle_replication_slot_timeout`,27`REPLICA IDENTITY FULL`, `binlog`, `gtid`, `server-id`, `LogMiner`, `change streams`,28`kafka-connect`, `connect-distributed.properties`, `consumer lag`, `auto.offset.reset`,29`enable.idempotence`, `isolation.level=read_committed`, `transactional.id`, `StreamsBuilder`,30`KTable`/`KStream`, `readStream`/`writeStream`, `withWatermark`, `checkpointLocation`,31`StreamExecutionEnvironment`, `KeyedProcessFunction`, `WatermarkStrategy`, `savepoint`,32`checkpoint`, `DLQ`/`dead letter`, and the phrases of the domain: "we want it in real time", "the33primary's disk filled up", "the connector has been down since Friday", "the deletes are missing",34"duplicate events are arriving", "events arrive out of order", "we have to reprocess the whole35history", "the source added a column and the consumer broke".3637**Not applicable**: see38- `data-platform-standards` (**mother**): **Kafka as an engine is hers** —topic partitions,39 retention, replicas, `min.insync.replicas`, KRaft— and so is the **schema registry** (Karapace,40 Apicurio, Confluent Schema Registry) and **PostgreSQL as an engine** with its replication, HA and41 PITR. Here, **what gets published on those topics, with what guarantees and what is done with the42 stream**. Her principle —**one store because it is needed, not because it is fashionable**— is43 inherited as **"one stream because it is needed, not because it is fashionable"** (§2.1).44- `message-brokers-standards` — **this is the most likely collision of the pair and45 the line is written here with precision**: **the broker as a component is hers** (Kafka, Pulsar,46 RabbitMQ, NATS: choice, topology, sizing, replication, retention, cluster operation, queues versus47 logs). **Processing the stream and capturing what goes into it belong here.** Mechanical cut-off48 rule: **if the broker makes the decision, it is hers; if the producer, the consumer or the49 processor makes it, it is ours.** "How many partitions can the cluster take?" is hers; "which50 partition key do I choose and what ordering does it guarantee me?" is ours.51- `microservices-architecture-standards` (**critical boundary; see §3.3, it is the most expensive52 design mistake in the domain**): **the outbox pattern, sagas, domain events and per-service data53 ownership are hers**. A **domain event** is designed and published by the application that owns54 the data; a **row change captured by CDC is not a domain event**. Here, the capture and transport55 mechanism — including CDC **over the outbox table**, which is its correct use.56- `lakehouse-standards` (**sister; boundary declared on both sides**): **what happens to the table57 when the changes land** is hers —*copy-on-write* versus *merge-on-read*, cost of58 `MERGE`/`DELETE`, files and deletion vectors, compaction, snapshots, catalog—. Here, **how the59 changes are captured and how they are processed on the way there**. **CDC is a typical lakehouse60 source, not the only one** (the lakehouse is fed by batch just the same) and **the lakehouse is61 not the only CDC destination** (a replica, a cache, a search index, another service). Neither62 presupposes the other.63- `data-engineering-standards` (**sister**): batch ingestion, orchestration, the watermark of an64 incremental, job idempotency, *backfill* and file format. She sets **when change capture is the65 answer**; here **how it is done well**. Her mirrored rule: the **file format** is hers, the66 **table format** belongs to `lakehouse-standards`.67- `data-governance-quality-standards` (**already on disk**): **data contracts, ownership, catalog,68 lineage and quality as a programme are hers**. Here, their execution in the stream: which schema69 gets published, who the responsible producer is and which assertions divert an event to the failed70 queue. Cut-off rule: if the question is "who is the owner and what does the contract promise?", it71 is hers; if it is "what does the consumer do when an event arrives that breaks it?", it is ours.72- `data-warehouse-modeling-standards`: the destination model. **A change stream is not an SCD2**:73 dimensional historisation is a modelling decision of hers; here the change is merely delivered.74- `privacy-engineering-standards`: **personal data, minimisation, retention and erasure are hers**.75 Here their technical constraint: a change published on a log with long retention **is one more76 copy of the personal data**, and a `DELETE` at the source **does not erase the previous events in77 the log** (§5).78- `observability-standards`: instrumentation, OTel, cardinality and the metrics backend. Here,79 **what to measure** in a stream (§6).80- `sre-practice-standards` (SLO, error budget, on-call), `incident-management-standards`,81 `backup-recovery-standards` (**the event log is not a backup**),82 `bcdr-standards`, `kubernetes-standards`, `iac-standards`, `cicd-standards`,83 `secrets-management-standards`, `identity-access-management-standards`,84 `grc-compliance-standards`, `mlops-standards`, `nosql-standards`, `search-engines-standards`,85 `vector-db-standards`, `graph-db-standards`, `timeseries-db-standards`,86 `oracle-dba-standards`/`sqlserver-dba-standards`/`mysql-mariadb-dba-standards` (**tuning and87 operating the source engine** are theirs; here only what CDC demands of it. Arbitration rule88 mirrored with `mysql-mariadb-dba-standards`: **capture is ours, impact on the engine is theirs**89 — the GTID format, the `binlog` as an engine mechanism and its purging are theirs; reading it as90 a change stream is ours), `ibm-i-rpg-standards` (**the Db2 for i equivalent, with the same91 mirrored rule**: the **journal and its receivers** —`CRTJRN`, `STRJRNPF`, `IMAGES(*BOTH)`,92 receiver management and purging— are hers, along with the platform's data traps (packed decimal,93 numeric dates, multi-member files, CCSID 65535); **reading that journal as a change stream is94 ours**. It is declared explicitly because **the mechanism catalogue in §2.2 did not include it**,95 and it must not be inferred from that that the platform has no log-based capture: it has one, and96 it is among the oldest), `aws-standards`/`azure-standards`/`gcp-standards` (MSK, DMS, Kinesis, Event Hubs,97 Pub/Sub, Datastream as managed services), `python-standards`, `jvm-spring-standards`98 (**Java/Kotlin code quality**: Flink and Kafka Streams are JVM and their *build*, tests and99 packaging are hers; stream design is ours), `scala-standards` (same boundary for100 the Scala API of Flink and Kafka Streams), `sql-standards` (Flink SQL and ksqlDB have101 streaming semantics of their own —windows, watermarks, dynamic tables— **which are ours**; the102 relational SQL written against a sink is hers).103104**Governing principle**: **streaming is not paid for in compute, it is paid for in continuous105operation.** A batch that fails is retried tomorrow; a stream that fails accumulates lag, holds106resources at the source and degrades while nobody is looking. Before adopting it, the question is107not "can I?", it is **"who watches it at 3 a.m. on a Sunday in August?"**.108109## 2. Default decisions110111> Verify version, licence and ownership on the web before committing to it in a real project (§8).112> In 2026 this subsector changed hands: **IBM completed the acquisition of Confluent on113> 17 Mar 2026**.114115### 2.1 The starting decision: do you really need streaming?116117Exhaust this ladder **before** introducing a stream. Every rung avoided is a permanent operation you118do not take on:119120| Stated need | Simplest solution | When it genuinely stops working |121|---|---|---|122| "We want it in real time" (for a report that is looked at in the morning) | **Incremental batch by watermark** (see `data-engineering-standards`) | Never stops working. This case **is not streaming** |123| Latency of minutes, moderate volume | **Incremental batch every 5-15 minutes** | When the source has no reliable watermark, or deletes must be captured |124| Query operational data with minimal delay | **Read replica** of the engine itself (see `data-platform-standards`) | When the destination is another engine, or another data model |125| You need **deletes**, ordering and the complete history of changes | **CDC** from the transaction log | — |126| Somebody or something **acts within seconds** on the event (fraud, stock, alerts, prices) | **Streaming** with stateful processing | — |127128Two questions that settle most arguments:1291301. **Which decision changes if the data arrives in 5 seconds instead of 15 minutes, and who makes131 it?** If the answer is "none" or "nobody", the real-time requirement does not exist.1322. **What is the cost of the stream being down for 4 hours on a Sunday?** If it is "none", you do133 not need streaming either. If it is "severe", you need **on-call**, not just technology.134135**Honest rule**: CDC is **the right answer to a problem other than real time**. Its real value is136usually **capturing deletes and rapid changes faithfully**, not latency. Many projects that "need137streaming" actually needed CDC dumping into a destination in micro-batches.138139### 2.2 How changes are captured: the mechanism matters more than the tool140141| Mechanism | How it works | Verdict |142|---|---|---|143| **Reading the transaction log** (PostgreSQL WAL, MySQL/MariaDB binlog, Oracle LogMiner, SQL Server CDC, MongoDB *change streams*, **Db2 for i journals and receivers** — see `ibm-i-rpg-standards`) | Reads the engine's write log: it sees **every** operation, in *commit* order, with before and after | **The correct way.** No schema impact, captures deletes, respects transactional ordering. **Absolute default** |144| **Triggers** writing to an audit table | The engine runs code on every write | **Vetoed unless the log is impossible.** Penalises **every** transaction of the operational system, is bypassed by badly written bulk operations, pollutes the schema and is forgotten when a new table is created |145| **Timestamp query** (`WHERE updated_at > :x`) | Periodic polling | **This is not CDC.** **It loses physical deletes** (for ever and silently) and **loses multiple changes between polls** (you only see the last state). Besides, `updated_at` is almost never reliable: skewed clocks, bulk updates that do not touch it, writes that commit out of order. Legitimate only as an **incremental batch**, knowing what is being lost |146| **Dual write** from the application (DB and then broker) | The app writes to both places | **FORBIDDEN** (§7): it is not atomic. See outbox in `microservices-architecture-standards` |147148**A warning about watermarks and commit ordering**: even with a perfect `updated_at`, a long149transaction may commit **after** your window has already passed its timestamp. Without window150overlap, that row is never ingested. It is the classic silent loss and it is one of the strongest151arguments in favour of the log.152153### 2.3 Toolchain154155| Area | Default | Verified status (Aug 2026) | Justifiable alternative |156|---|---|---|---|157| CDC capturer | **Debezium 3.6.x** | 3.6.0.Final (1 Jul 2026); 3.7.0.Alpha1 in progress. **Apache 2.0** (verified in the repository). De facto standard, backed by Red Hat | **Flink CDC** (Apache 2.0) if processing is already Flink and you want a single system |158| Running the capturer | **Debezium Server** (standalone) if you do not need Kafka Connect | Exists and is maintained; in the 3.6 series it is **migrating from the old embedded engine to the Quarkus extension** (`debezium-quarkus-engine`), with lifecycle control for *hot standby* with external locking. **Careful: it is an architectural migration in progress, not a consolidated mode** | **Distributed Kafka Connect** when you already operate Connect: it remains the most proven route and the one that has had the most eyes on it |159| Broker | **Kafka** (the component and its operation belong to `data-platform-standards` / `message-brokers-standards`) | Kafka 4.3.1 (Jun 2026), Apache 2.0, ASF. **IBM's acquisition of Confluent does not change Kafka's licence**: the project belongs to the ASF and its PMC is independent | — |160| Demanding stateful processing | **Apache Flink 2.3.0** | 2.3.0 stable according to the official downloads page (Jun 2026); 2.2.1, 2.1.3 and 1.20.5 also published. **It does not lose the technical crown, but it does lose new teams** because of its operational complexity: it demands dedicated people | **Managed Flink service** if you have no platform |161| Processing embedded in a JVM service | **Kafka Streams** | Alive. No cluster of its own; **bounded by the number of partitions** and with less sophisticated time/late-data handling than Flink | — |162| Processing where Spark already exists | **Spark Structured Streaming** | Alive (Spark 4.2.0, Jul 2026). Micro-batches: latency of seconds, not milliseconds. Wins by inertia if the team is already Spark | — |163| Managed SQL-first engines (RisingWave, Materialize, Decodable, ksqlDB…) | — | They greatly reduce the operational load in exchange for flexibility and for lock-in | Valid with an ADR and an exit plan |164| Event format | **Avro or Protobuf with a schema registry** | The registry belongs to `data-platform-standards` | JSON only with validation in CI |165166**Criterion for choosing the processor, one line each**:167- **Flink** if there is **large state, serious event-time semantics and late data that matters**,168 and you have somebody to operate it.169- **Kafka Streams** if it is a **JVM application that already lives inside Kafka** and you do not170 want another cluster.171- **Spark Structured Streaming** if **you already operate Spark** and second-level latency is172 enough.173- **None of them** if what you needed was an incremental batch (§2.1). It is the right answer more174 often than is admitted.175176**Retired or declining engines — do not choose them for anything new**: classic Spark Streaming177(DStreams) is obsolete compared with Structured Streaming; Apache Apex has had no development since1782019; Samza has not released since 2023; Storm is maintained but is rarely the right choice today.179180**Ownership and licence risk**: **IBM completed the purchase of Confluent on 17 Mar 2026** (around181$11bn; WarpStream included). Kafka as a project is unaffected, but **the Confluent Community182License** —which covers Schema Registry, REST Proxy, ksqlDB and several connectors— **is not OSI183open source** and forbids offering them as a competing service. With the new owner, the pricing and184support model is a risk to watch, not a settled fact (§8). Practical consequence: **prefer Karapace185or Apicurio** for the registry (criterion already set in `data-platform-standards`) and record in186the ADR which non-OSI-licensed components you depend on.187188## 3. Structure and conventions189190### 3.1 CDC in practice: the initial snapshot and resuming191192- **The initial snapshot is the most expensive and most dangerous moment of the whole project**:193 reading an entire table on a production system. Rules:194 - **Read from a replica** whenever the mechanism allows it; if not, plan it in a low-load window,195 with bounded reads and `statement_timeout`.196 - **Prefer the incremental snapshot** (chunked, resumable, in parallel with the stream) over the197 blocking one. A snapshot that cannot be resumed turns into five twelve-hour retries.198 - **Measure and budget how long it takes before running it in production.** It is the figure that199 decides whether the project is viable.200 - **Re-snapshotting has to be a planned and rehearsed operation** (signals / *ad hoc snapshot*),201 not an accident: you will need it when a slot is invalidated or a position is lost.202- **Resuming after a crash**: the capturer stores its **offset** (LSN, GTID, *resume token*)203 durably and outside its own process. Rules: the offset store goes into the backup; **if it is204 lost, the price is a full new snapshot**; and **the offset is committed after delivering, never205 before** — the other way round loses changes silently, this way they merely duplicate (and that206 is what idempotency is for, §3.4).207- **Heartbeats**: in databases with many tables where you only capture a few, and in databases with208 little activity, the capturer may not advance its offset for hours while the WAL piles up.209 **Enable heartbeats**; it is the specific mitigation and almost nobody knows about it.210- **Filter at the source, not at the destination**: include only the necessary tables and columns211 (bounded publications, include lists). Less load, less personal-data surface (§5), less schema212 noise.213214### 3.2 The impact on the source: **the production failure par excellence**215216**A logical replication slot retains WAL for as long as its consumer does not confirm. If the217consumer stops and nobody notices, `pg_wal` grows until the disk fills and the primary goes down.**218This is not a rare case: it is *the* incident of this domain. Verified against PostgreSQL 18:219220- **`max_slot_wal_keep_size`** (since PG 13; reloadable via SIGHUP, in MB if no unit is given)221 limits the WAL that slots may retain. With the default value **`-1` retention is unlimited**,222 which is exactly the configuration that fills the disk. **Always set it to a value consistent with223 the real free space.**224- **The trade-off has to be accepted explicitly**: when the limit is exceeded, PostgreSQL225 **invalidates the slot** and recycles the WAL. The consumer can no longer continue and **a new226 slot and, typically, a full initial snapshot will be needed**. It is a conscious decision: **you227 sacrifice CDC to save the database**. It is the right choice, and it has to be written in the228 runbook before it happens.229- **`idle_replication_slot_timeout`** (**new in PostgreSQL 18**; unitless value in seconds, `0`230 disables it) automatically invalidates inactive slots. **Invalidation happens at the231 *checkpoint***, so there is a delay between crossing the threshold and the actual invalidation. It232 does not apply to slots that do not reserve WAL nor to slots synchronised from the primary233 (`synced = true`). Reasonable values are of the order of 48-72 h.234- **Mandatory monitoring of `pg_replication_slots`**: `active`, `wal_status`, `safe_wal_size`235 (with a positive `max_slot_wal_keep_size` it indicates how much WAL is left before invalidation;236 **negative means it has already been passed**) and `invalidation_reason`. **Alert with a threshold237 on slot WAL retention and on inactive slots, without exception.** And **delete orphan slots**: a238 slot from a test that nobody removed is a time bomb.239- **Long transactions at the source** make everything worse: they retain WAL and can fill the240 storage on their own. Complement with `wal_sender_timeout` (60 s by default) to detect dead241 consumers.242- **`REPLICA IDENTITY`**: by default PostgreSQL only publishes the primary key in the "before" of an243 `UPDATE`/`DELETE`. `REPLICA IDENTITY FULL` gives the full row **at the cost of more WAL and more244 load**: enable it per table and only if the consumer needs it, never globally.245- **Other engines, same principle**: in MySQL/MariaDB, binlog retention (`binlog_expire_logs_seconds`)246 sets your resume window — if the consumer is down for longer than that retention, **the position247 is lost and a re-snapshot is required**. In Oracle, log mining has a cost in the engine itself.248 **Coordinate the parameter with the engine's DBA, it is a shared decision.**249- **Cross-cutting rule**: CDC **couples the health of your analytics platform to the health of the250 operational database**. It is the hidden cost of the technique and it has to be told in writing to251 the team that owns the source before anything is switched on.252253### 3.3 CDC versus outbox — **the boundary and the most expensive design mistake in the domain**254255**Say it plainly and without lukewarm qualifiers: table changes captured by CDC are NOT domain256events.** An `UPDATE` on `orders` tells you that some columns changed value; it does **not** tell you257that the order was cancelled, nor why, nor by whom. Publishing raw CDC as a contract between258services means:259260- **You couple every consumer to the internal physical schema** of the source service. From that261 moment on, that service can no longer refactor its database: any `ALTER TABLE` is a breaking change262 in a public API that nobody declared as such. It is exactly the same mistake as letting another263 service query your database, with an intermediate hop that disguises it.264- **You lose intent.** The row diff does not contain the "why". Each consumer reconstructs the265 semantics its own way, and all the reconstructions are different and plausible.266- **You expose everything that is in the table**, including columns that should never have left the267 service (§5).268- **No contract and no versioning**: a refactor breaks consumers in production without prior269 warning.270271**The correct division, and the boundary with `microservices-architecture-standards`**:272273| Case | Correct pattern | Owner of the decision |274|---|---|---|275| Publish that **something of business significance happened** so other services react | **Outbox**: the application writes the domain event, with its schema and its version, in the same transaction as the state change | `microservices-architecture-standards` |276| **Transport** that event reliably from the outbox table to the broker | **CDC over the outbox table** — this is the correct and virtuous use of CDC in service architecture | Here |277| Feed an **analytical store, a lakehouse, a cache or a search index** from existing tables | **Direct CDC over the tables**, with the destination treated as an **internal** consumer | Here |278| Integrate a **legacy system** that cannot be modified | **Direct CDC**, with a **mandatory anti-corruption layer** translating to a contract of your own before exposing it to anybody else | Here |279280**Hard rules that follow**:281- **A raw CDC topic is never a public contract.** If somebody outside the owning team consumes it,282 either it becomes a domain event with an outbox, or an anti-corruption layer with its own versioned283 schema is placed in between. No exceptions.284- The cost of the outbox is real and must be stated: **it touches every write path, the table grows285 and has to be purged, and you maintain two schemas** (the table's and the event's). It is paid286 because the contract is worth it.287- **Tombstones**: on a compacted topic, a delete is represented by a message with a null value. If288 your consumer does not handle them, **deletes are not propagated** and your copy accumulates289 ghosts. It is a silent failure, which is the worst kind.290291### 3.4 Delivery semantics: what they really mean292293| Semantics | What it guarantees | How it is achieved | Verdict |294|---|---|---|---|295| **At-most-once** | Never duplicates; **may lose** | Commit the offset **before** processing | **It is almost always an accident, not a design.** And it happens through carelessness |296| **At-least-once** | Never loses; **may duplicate** | Commit the offset **after** processing | **The correct default.** Everything else is built on top |297| **Exactly-once** | Each effect is applied once | Transactions and state coordinated **inside the system** | **Real, but far narrower than it is sold as** (see below) |298299**What "exactly once" really means**: it is **exactly-once *of effect***, not of delivery, and300**only within the boundary of the system that implements it** — reading from Kafka, updating state301and writing to Kafka in one transaction (Kafka Streams / transactional producer with302`read_committed`), or Flink's *checkpoint* mechanism with two-phase transactional sinks. **As soon303as an effect leaves that boundary** —calling an API, sending an email, writing to a database that304does not take part in the transaction— **the guarantee is over**. It also costs latency and305operational complexity.306307**Therefore: consumer idempotency is the only real defence, and it is mandatory.** It is not an308alternative to *exactly-once*, it is its prerequisite and its substitute in 90 % of cases. It is309implemented with:310- **Business key + upsert** (`INSERT ... ON CONFLICT`, `MERGE`) instead of a plain `INSERT`.311- **Deduplication by event identifier** with a bounded window, when the upsert does not apply.312- **Version / sequence number comparison**: discard the event whose `lsn`/`sequence` is lower than313 or equal to the one already applied. This also protects against reordering.314- **Non-idempotent external effects** (charging, sending, notifying) **protected with an idempotency315 key and an execution log**. A reprocess must not bill twice.316317### 3.5 Ordering and partitioning318319- **Ordering is only guaranteed within a partition.** There is no global ordering in a partitioned320 log, and whoever assumes there is builds a system that fails intermittently and randomly.321- **Choosing the partition key is choosing what is ordered.** Rule: **the key is the entity whose322 sequence of changes matters** — typically the primary key of the source row. That way all the323 changes of the same order go to the same partition and arrive in order.324- **Consequences that have to be accepted**:325 - **Partition skew**: a very active entity saturates its partition and limits parallelism. It is326 detected by measuring lag **per partition**, not the aggregate.327 - **Changing the number of partitions breaks key affinity** and therefore historical ordering.328 It is a design decision with consequences, not a capacity adjustment (topic operation belongs to329 `data-platform-standards`).330 - **Consumer parallelism is bounded by the number of partitions.**331- **If ordering between different entities matters** (a transfer between two accounts), partitioning332 by entity does **not** give it to you. Either you put both in the same partition via a composite333 business key, or you solve it with state and event time in the processor, or you redesign the334 operation as a single event. Choose explicitly: do not leave it to chance.335- **Reordering across topics**: two different streams have no relative ordering. If a consumer needs336 "customer first, then order", that is a **dependency** to be resolved in the consumer (wait, look337 up, retry), not a transport guarantee.338339### 3.6 Schema and evolution340341The schema registry and its compatibility configuration belong to `data-platform-standards`. What is342specific to the stream:343344- **Backward compatibility** (a new consumer reads old data) as the mandatory minimum;345 **forward** (an old consumer reads new data) when you do not control the consumers, which is the346 usual case. Verified in CI, not in somebody's head.347- **What to do when the source adds a column**: it is **additive and compatible** if the field has a348 default value. Decide explicitly whether it is propagated (**not everything that appears at the349 source should leave the source**, §5). If your connector has a column include list, the new column350 does not get in by itself: that is the safe option.351- **What to do when the source drops or renames a column**: it is **incompatible** and breaks352 consumers. It is handled as an *expand/contract* migration, coordinated with the team that owns353 the source: publish both fields, migrate consumers, retire the old one. **That the capturer354 tolerates the change does not mean your consumers do.**355- **Type change**: almost always incompatible. New field + coexistence + retirement.356- **Events already published cannot be rewritten.** A consumer reprocessing from the beginning of357 the log will see **every** historical schema: your consuming code must be able to read them all358 for as long as the log retains them. This is not theoretical, it is what happens on the day you359 reprocess.360- **The owner of the event schema is the producer**, with a published and versioned contract. A361 schema change without notice is an incident, not a deployment.362363### 3.7 Processing in motion: time, windows and state364365- **Event time versus processing time**: **event time** (when it happened) is the correct one for366 anything the business is going to read; **processing time** (when the engine saw it) produces367 **non-reproducible** results: reprocessing gives a different result from the original. Rule:368 **always aggregate by event time**; use processing time only for telemetry of the system itself.369- **Watermarks**: the engine's assertion that "I no longer expect events earlier than T". It is370 **a heuristic, not a fact**, and it embodies the central trade-off of the domain: **aggressive371 watermark = fast results and more data discarded; permissive watermark = late results and more372 state retained**. It is chosen with real lateness data from the stream, documented and reviewed.373- **Data arriving late**: decide and write down what happens to it. Three legitimate options —374 discard it (counting it, always), accept it with an allowed lateness and **re-emit** the corrected375 result, or divert it to a side stream for separate handling. **The illegal option is not376 deciding**, because the silent default is to discard it without counting it, and then the figures377 do not add up and nobody knows why.378- **Windows**: *tumbling* by default; *sliding* multiply state and compute by the overlap; session379 windows when the business talks about sessions. **Every open window is retained state**: a 30-day380 window over a high-cardinality key is a memory problem, not a requirement.381- **State**: it is the most expensive and most fragile asset of a stream. Rules:382 - **TTL on all keyed state**, without exception. Keyed state with no expiry grows until it brings383 the job down, and it does so months after nobody remembers the design.384 - **Checkpoints/savepoints on durable storage**, backed up and **with a tested restore**. A job385 whose state cannot be restored is not recoverable: only reprocessable.386 - **State compatibility limits deployment**: changing the logic or the state schema may prevent387 restoring from the previous *savepoint*. Decide in advance whether a deployment keeps state or388 starts from scratch — and if it starts from scratch, how long it takes to catch up.389 - **Stateful *joins* over streams** require retaining both sides: define the window or state390 explodes.391392## 4. Quality and testing — gates393394In increasing order of cost. **Those marked as gates break the build or the deployment.**3953961. **Capturer and job configuration as code**, versioned and reviewed: connectors, table and column397 include lists, snapshot mode, partition keys, retention, window parameters. Nothing created by398 hand against a production API. *CI gate.*3992. **No secrets in the connector configuration**: credentials come from the secrets manager400 (see `secrets-management-standards`). A connector file with the production database password is a401 finding, not an oversight. *CI gate.*4023. **Schema compatibility verified in CI** against the registry, with the declared policy403 (minimum `BACKWARD`). An incompatible change **breaks the build**. *Gate.*4044. **Unit tests of the processing logic** with fixed events and an expected result, covering the405 happy path **and the mandatory edges**: **duplicate event**, **out-of-order event**,406 **delete / tombstone**, **null field**, **late event beyond the watermark**,407 **empty window**, **brand-new key never seen before**, **restart mid-window**. *CI gate.*4085. **End-to-end idempotency test**: delivering the same batch of events twice produces the same409 state at the destination (same count, same checksum). **Without this test, idempotency is an410 intention.** *CI gate.*4116. **Delete propagation test**: delete at the source and check that the destination reflects the412 delete. It is the most expensive silent failure and the least tested. *Gate.*4137. **Resume test**: kill the capturer or the job, start it and check that it **neither loses nor414 duplicates in a non-idempotent way**. With containers of the real engine (not with doubles),415 because the log's behaviour is what is being tested.4168. **State restore test** from *checkpoint*/*savepoint*, and of **changing the application version**417 while keeping state. It is done before the first production deployment, not after the first418 incident.4199. **Periodic reconciliation against the source** (count and sum of a key measure, with a declared420 tolerance), scheduled. **A CDC stream diverges from the source over time; without reconciliation421 you do not find out.** This is the most important operational gate of them all.42210. **Reprocessing rehearsal**: reprocess from the beginning of the log in a mirror environment,423 measuring how long it takes and what it costs. If it does not fit in the time available, your424 recovery plan does not exist.42511. **Dependencies pinned by hash/digest** in connectors, images and job libraries (§5).426 *CI gate.*427428## 5. Stack security429430- **CDC copies everything that is in the table, including the columns that should never have left.**431 Filter **at the capturer** (column list, bounded publication, masking in the connector), not432 downstream: every PII column you do not capture is a retention, erasure and breach problem you433 will not have. Coordinate with `privacy-engineering-standards`.434- **An event log with long retention is one more copy of the personal data**, with its own435 classification, its declared retention and its access control. **Infinite retention "just in case"436 is forbidden** (§7).437- **The right to erasure clashes with the log**: a `DELETE` at the source generates a new event,438 **it does not erase the previous events**. A compacted topic with a *tombstone* removes the439 previous versions of that key **over time, not immediately**, and only if it is compacted. If440 there is an erasure commitment, **topic retention becomes a compliance parameter**, or it is441 designed with pseudonymisation / *crypto-shredding* from the start. The policy belongs to442 `privacy-engineering-standards`; **the technical constraint and the parameter are set here**.443- **Least privilege at the source**: the capturer's user has **only** the replication/read444 permissions it needs, on the agreed tables. **It is not a superuser**, even if the connector445 tutorial says so.446- **The capturer authenticates with rotatable, short-lived credentials**; no eternal static447 passwords in a connector file. TLS mandatory against the source and against the broker;448 authorisation per topic and per consumer group.449- **PII out of logs and error messages**: a connector that logs the row it could not serialise is a450 leak. Log key and offset, not content. The same applies to the **dead-letter queue**: it is a data451 store with real content, and it needs its own access control and its own retention.452- **Supply chain — verified precedents, not hypotheses**: **Trivy** (Mar 2026), **LiteLLM**453 on PyPI (1.82.7/1.82.8, 24 Mar 2026), **Telnyx** (Mar 2026), **Microsoft's `durabletask`**454 (1.4.1-1.4.3, May 2026) and **`elementary-data` 0.23.3** (Apr 2026, with the `.pth` pattern that455 runs when the interpreter starts). The current generation, **"Mini Shai-Hulud"456 (CVE-2026-45321)**, active since late Apr 2026, propagates between npm and PyPI, **extracts OIDC457 tokens from the memory of the GitHub Actions runner** and has gone as far as **forging SLSA458 level 3 provenance attestations**. Mandatory consequence: pin by hash/digest everything that runs459 in a process holding credentials for the source database or the broker; a quarantine window of460 days before adopting newly published versions; **provenance attestation is no longer sufficient461 proof on its own**; and a *runner* does not hold credentials for more than one environment.462463## 6. Performance and operability464465### 6.1 Consumer lag is the primary metric466467- **Measure lag in time, not just in number of messages.** "12,000 messages behind" says nothing;468 "18 minutes behind" does. And **measure it per partition**: the aggregate hides skew,469 which is the most frequent cause.470- **Alert on trend, not on an isolated threshold**: lag that grows steadily is an incident even if471 its absolute value is still small. The fixed threshold warns when there is no margin left.472- Mandatory complementary signals: **capturer lag relative to the source** (which is different from473 consumer lag), **WAL retention / slot size** (§3.2), job restart rate, **retained state**,474 *checkpoint* duration and failures, message rate to the DLQ,475 and **data age at the destination** (freshness, which is the signal the business cares about;476 see `data-engineering-standards` §6).477- **Actionable alert with a runbook**, always: what broke, what depends on it, how it is recovered,478 how long it takes. A lag alert with no runbook at 4 a.m. is useless.479- **If there is a latency commitment, there is on-call.** An SLO of seconds with nobody to respond480 out of hours is a documented lie (criterion inherited from `data-engineering-standards` §6.3 and481 `sre-practice-standards`).482483### 6.2 Reprocessing, DLQ and the question that decides the architecture484485- **The dead-letter queue (DLQ) is mandatory and has an owner.** Rules: every diverted message486 carries **the cause and the original offset**; there is a written reinjection procedure; and487 **the DLQ is monitored with an alert**. A DLQ nobody looks at is a dump of lost data with the488 appearance of a solution. A poison message must **never** be retried infinitely: it blocks the489 whole partition and stops the stream.490- **Reprocessing from the beginning**: it is the capability that makes a stream operable —491 fixing a logic error, populating a new destination, recovering from corruption. Requirements:492 **idempotent consumers** (§3.4), a destination that tolerates rewriting, capacity to absorb the493 burst without bringing anything downstream down, and **a prior measurement of how long it takes**.494 Rehearse it (§4).495- **The question that decides the architecture: how long do you keep the log?** It is not an496 operational parameter, it is a structural decision:497498| Retention | What it buys you | What it costs you |499|---|---|---|500| **Short** (hours/days) | Cheap storage, small personal-data surface | **You can only reprocess the recent past.** A new destination has to be seeded from the source with a snapshot |501| **Medium** (weeks) | Realistic reprocessing of incidents | Cost and data classification |502| **Long or infinite** (log as the source of truth) | Rebuild any destination at any time | Growing cost, **direct conflict with retention and erasure of personal data**, and **every historical schema alive for ever** in your consuming code |503504 **Decide it in writing in the ADR, with the privacy team present**, and be clear that **the log505 is not a backup**: it has neither the recovery model, nor the verification, nor the isolation of506 one (see `backup-recovery-standards`).507508### 6.3 Cos509510…(truncated)