Replication Patterns
Coverage
The catalog of replication topologies and the operational discipline that makes them work in production. Covers the three foundational topologies (single-leader / primary-replica, multi-leader / multi-primary, leaderless / quorum), the synchrony spectrum (sync, semi-sync, async, quorum-sync), the mechanism choices (statement-based, row-based, trigger-based, logical, physical), the read-after-write consistency problem and its mitigations (sticky session, read-from-leader, monotonic reads, version tokens), the failover model and quorum-based split-brain prevention, the conflict-resolution choices in multi-leader and leaderless systems (LWW, CRDTs, vector clocks, application merge), and the relationship to the CAP/PACELC choices the topology realizes.
Philosophy
Replication is the discipline that gives a database fault tolerance, read scaling, and disaster recovery — at the cost of consistency-handling, conflict resolution, and operational complexity.
The default starting point is single-leader with asynchronous replication: simple, well-understood, sufficient for most read-mostly workloads. The departures from this default — multi-leader, leaderless, synchronous, geographic — each address a specific requirement (multi-region writes, strong consistency under partition, RPO=0) and add proportional complexity.
The most common production failures are not in the replication topology itself but in the application's handling of its consequences: stale reads producing user-visible bugs, split brain producing silent data divergence, failover producing unrehearsed surprises. The discipline is treating replication as an architecture the application is co-designed with, not a database feature that handles itself.
Topology Selection
| Topology |
Best for |
Trade-offs |
| Single-leader, async |
Read-heavy workloads with tolerance for stale reads |
Replication lag; possible data loss on leader failure |
| Single-leader, sync |
Workloads requiring RPO=0 |
Write latency; replica failure can block writes |
| Multi-leader |
Multi-region writes; active-active disaster recovery |
Conflict resolution complexity |
| Leaderless (quorum) |
High availability with tunable consistency |
Read latency = slowest of R; quorum-sizing decisions |
| Synchronous via Raft/Paxos (Spanner, CockroachDB) |
Strong consistency at scale |
High write latency for distant replicas |
The starting point for most workloads is single-leader async; depart from this default only when the workload requires it.
Synchronous vs Asynchronous Trade-off
| Property |
Synchronous |
Asynchronous |
| Write latency |
Higher (waits for replica) |
Lower (acks immediately) |
| RPO (data loss on failure) |
Zero |
Up to lag window |
| Read consistency from replicas |
Strong |
Eventual |
| Replica failure impact |
Can block writes |
None |
| Use case |
High-stakes financial transactions, strong-RPO systems |
Most production read-replicas |
Semi-sync (wait for one replica with timeout-to-async fallback) is the middle ground; production-default for many systems.
Read-After-Write Mitigations
| Mitigation |
How it works |
Cost |
| Read-from-leader for a window after write |
Client routes back to leader for N seconds |
Loses read scaling for write-heavy users |
| Sticky session |
Client always reads from same replica |
Replica failure invalidates session |
| Monotonic reads |
Client tracks last-seen version; replica must be ≥ that fresh |
Requires version-token plumbing |
| Wait-for-replica |
Client waits until replica catches up |
Adds latency to the read |
| Accept stale reads |
Application tolerates staleness |
Only viable when stale data is OK |
Every read-mostly workload with async replicas must choose one. Defaulting to "read from any replica" produces stale-data bugs.
Failover and Split-Brain Prevention
| Mechanism |
Split-brain risk |
Used by |
| Manual operator failover |
None (if procedure is correct) |
Small / legacy systems |
| Heuristic auto-failover (no quorum) |
High under partition |
Older Postgres tools (without proper consensus) |
| Quorum-based promotion (Raft / Paxos) |
Eliminated by majority requirement |
Modern HA tools (Patroni, etcd, CockroachDB internal) |
| STONITH fencing |
Old leader killed; cannot continue writing |
Pacemaker / Corosync HA |
A system that must not split-brain uses quorum-based promotion. The cost is requiring an odd number of voting nodes ≥ 3.
Verification
After applying this skill, verify:
Do NOT Use When
| Instead of this skill |
Use |
Why |
| Horizontally partitioning data across nodes |
sharding-strategy |
sharding partitions data; replication copies it |
| Reasoning about the CAP/PACELC theoretical frame |
cap-theorem-tradeoffs |
CAP names the trade-off; this skill realizes it |
| Single-node transactional guarantees |
acid-fundamentals |
ACID is the single-system frame |
| Choosing isolation levels |
transaction-isolation |
transaction-isolation owns concurrency; this owns multi-node |
| Indexing |
indexing-strategy |
indexing is within-node retrieval |
| Tuning a slow query |
query-optimization |
query-optimization is per-query |
Key Sources
- Kleppmann, M. (2017). Designing Data-Intensive Applications. O'Reilly. Chapter 5 (Replication) — modern comprehensive treatment of all three topologies with practical detail.
- Lamport, L. (1998). "The Part-Time Parliament". ACM TOCS, 16(2). The Paxos consensus algorithm; foundation of synchronous replication with split-brain prevention.
- Ongaro, D., & Ousterhout, J. (2014). "In Search of an Understandable Consensus Algorithm (Raft)". USENIX ATC 2014. The Raft consensus algorithm; basis of etcd, CockroachDB, many modern HA systems.
- DeCandia, G., Hastorun, D., Jampani, M., et al. (2007). "Dynamo: Amazon's Highly Available Key-Value Store". SOSP 2007. The foundational paper on leaderless (Dynamo-style) replication; basis of Cassandra, Riak, DynamoDB.
- PostgreSQL Global Development Group. "PostgreSQL Documentation — Replication" and "Logical Replication". Reference for Postgres physical streaming and logical replication.
- MySQL Reference Manual. "Replication". MySQL replication including binlog formats, semi-sync, and group replication.
- MongoDB Manual. "Replication". MongoDB replica sets, automatic failover, and oplog-based replication.
- Vogels, W. (2009). "Eventually Consistent". ACM Queue. Practitioner essay on eventual consistency in replicated systems.
- Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). "A Comprehensive Study of Convergent and Commutative Replicated Data Types". CRDT foundational paper; basis for multi-leader conflict resolution without data loss.
- Brewer, E. (2012). "CAP Twelve Years Later". IEEE Computer. Brewer's retrospective on CAP, useful for understanding the consistency-availability trade-offs replication realizes.
1---2name: replication-patterns3description: Use when designing how a database keeps multiple copies of its data in agreement across nodes for availability, read scaling, and disaster recovery: the three foundational topologies (single-leader / primary-replica, multi-leader / multi-primary, leaderless / quorum), synchronous vs asynchronous replication and the replication-lag trade-off, log shipping vs statement replication vs trigger-based replication, the read-after-write consistency problem and its mitigations (sticky session, read-from-leader, monotonic reads), the failover model and split-brain risk, and the relationship to the CAP/PACELC choices the topology realizes. Do NOT use for horizontal partitioning across nodes (use sharding-strategy), the CAP theoretical frame itself (use cap-theorem-tradeoffs), single-node transactional guarantees (use acid-fundamentals), or query tuning (use query-optimization).4license: MIT5---6
7# Replication Patterns
8
9## Coverage
10
11The catalog of replication topologies and the operational discipline that makes them work in production. Covers the three foundational topologies (single-leader / primary-replica, multi-leader / multi-primary, leaderless / quorum), the synchrony spectrum (sync, semi-sync, async, quorum-sync), the mechanism choices (statement-based, row-based, trigger-based, logical, physical), the read-after-write consistency problem and its mitigations (sticky session, read-from-leader, monotonic reads, version tokens), the failover model and quorum-based split-brain prevention, the conflict-resolution choices in multi-leader and leaderless systems (LWW, CRDTs, vector clocks, application merge), and the relationship to the CAP/PACELC choices the topology realizes.
12
13## Philosophy
14
15Replication is the discipline that gives a database fault tolerance, read scaling, and disaster recovery — at the cost of consistency-handling, conflict resolution, and operational complexity.
16
17The default starting point is single-leader with asynchronous replication: simple, well-understood, sufficient for most read-mostly workloads. The departures from this default — multi-leader, leaderless, synchronous, geographic — each address a specific requirement (multi-region writes, strong consistency under partition, RPO=0) and add proportional complexity.
18
19The most common production failures are not in the replication topology itself but in the application's handling of its consequences: stale reads producing user-visible bugs, split brain producing silent data divergence, failover producing unrehearsed surprises. The discipline is treating replication as an architecture the application is co-designed with, not a database feature that handles itself.
20
21## Topology Selection
22
23| Topology | Best for | Trade-offs |
24|---|---|---|
25| Single-leader, async | Read-heavy workloads with tolerance for stale reads | Replication lag; possible data loss on leader failure |
26| Single-leader, sync | Workloads requiring RPO=0 | Write latency; replica failure can block writes |
27| Multi-leader | Multi-region writes; active-active disaster recovery | Conflict resolution complexity |
28| Leaderless (quorum) | High availability with tunable consistency | Read latency = slowest of R; quorum-sizing decisions |
29| Synchronous via Raft/Paxos (Spanner, CockroachDB) | Strong consistency at scale | High write latency for distant replicas |
30
31The starting point for most workloads is single-leader async; depart from this default only when the workload requires it.
32
33## Synchronous vs Asynchronous Trade-off
34
35| Property | Synchronous | Asynchronous |
36|---|---|---|
37| Write latency | Higher (waits for replica) | Lower (acks immediately) |
38| RPO (data loss on failure) | Zero | Up to lag window |
39| Read consistency from replicas | Strong | Eventual |
40| Replica failure impact | Can block writes | None |
41| Use case | High-stakes financial transactions, strong-RPO systems | Most production read-replicas |
42
43Semi-sync (wait for one replica with timeout-to-async fallback) is the middle ground; production-default for many systems.
44
45## Read-After-Write Mitigations
46
47| Mitigation | How it works | Cost |
48|---|---|---|
49| Read-from-leader for a window after write | Client routes back to leader for N seconds | Loses read scaling for write-heavy users |
50| Sticky session | Client always reads from same replica | Replica failure invalidates session |
51| Monotonic reads | Client tracks last-seen version; replica must be ≥ that fresh | Requires version-token plumbing |
52| Wait-for-replica | Client waits until replica catches up | Adds latency to the read |
53| Accept stale reads | Application tolerates staleness | Only viable when stale data is OK |
54
55Every read-mostly workload with async replicas must choose one. Defaulting to "read from any replica" produces stale-data bugs.
56
57## Failover and Split-Brain Prevention
58
59| Mechanism | Split-brain risk | Used by |
60|---|---|---|
61| Manual operator failover | None (if procedure is correct) | Small / legacy systems |
62| Heuristic auto-failover (no quorum) | High under partition | Older Postgres tools (without proper consensus) |
63| Quorum-based promotion (Raft / Paxos) | Eliminated by majority requirement | Modern HA tools (Patroni, etcd, CockroachDB internal) |
64| STONITH fencing | Old leader killed; cannot continue writing | Pacemaker / Corosync HA |
65
66A system that must not split-brain uses quorum-based promotion. The cost is requiring an odd number of voting nodes ≥ 3.
67
68## Verification
69
70After applying this skill, verify:
71- [ ] The replication topology is intentional and documented: single-leader / multi-leader / leaderless; sync / async / semi-sync; physical / logical / trigger-based.
72- [ ] Read-after-write consistency is handled explicitly. Sticky session, read-from-leader, monotonic reads, version tokens, or accept-stale is a named choice.
73- [ ] Replication lag is monitored. Lag thresholds trigger alerts before users notice.
74- [ ] Failover procedure is documented, tested, and rehearsed. Untested failover is failover that fails first time.
75- [ ] Split-brain prevention uses quorum-based promotion for any system requiring it. Heuristic failover is recognized as risky.
76- [ ] Backups exist separately from replication. Replica state and backup state are distinguished.
77- [ ] For multi-leader: the conflict resolution mechanism (LWW / CRDT / vector clocks / app merge) is defined and tested with concurrent-write scenarios.
78- [ ] For leaderless: W and R values are chosen for the workload's consistency-vs-latency target; W+R>N if strong consistency is required.
79- [ ] Cross-region replication latency is measured and accepted as part of the design budget.
80
81## Do NOT Use When
82
83| Instead of this skill | Use | Why |
84|---|---|---|
85| Horizontally partitioning data across nodes | `sharding-strategy` | sharding partitions data; replication copies it |
86| Reasoning about the CAP/PACELC theoretical frame | `cap-theorem-tradeoffs` | CAP names the trade-off; this skill realizes it |
87| Single-node transactional guarantees | `acid-fundamentals` | ACID is the single-system frame |
88| Choosing isolation levels | `transaction-isolation` | transaction-isolation owns concurrency; this owns multi-node |
89| Indexing | `indexing-strategy` | indexing is within-node retrieval |
90| Tuning a slow query | `query-optimization` | query-optimization is per-query |
91
92## Key Sources
93
94- Kleppmann, M. (2017). *Designing Data-Intensive Applications*. O'Reilly. Chapter 5 (Replication) — modern comprehensive treatment of all three topologies with practical detail.
95- Lamport, L. (1998). ["The Part-Time Parliament"](https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf). *ACM TOCS*, 16(2). The Paxos consensus algorithm; foundation of synchronous replication with split-brain prevention.
96- Ongaro, D., & Ousterhout, J. (2014). ["In Search of an Understandable Consensus Algorithm (Raft)"](https://raft.github.io/raft.pdf). *USENIX ATC 2014*. The Raft consensus algorithm; basis of etcd, CockroachDB, many modern HA systems.
97- DeCandia, G., Hastorun, D., Jampani, M., et al. (2007). ["Dynamo: Amazon's Highly Available Key-Value Store"](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf). *SOSP 2007*. The foundational paper on leaderless (Dynamo-style) replication; basis of Cassandra, Riak, DynamoDB.
98- PostgreSQL Global Development Group. ["PostgreSQL Documentation — Replication"](https://www.postgresql.org/docs/current/runtime-config-replication.html) and ["Logical Replication"](https://www.postgresql.org/docs/current/logical-replication.html). Reference for Postgres physical streaming and logical replication.
99- MySQL Reference Manual. ["Replication"](https://dev.mysql.com/doc/refman/8.0/en/replication.html). MySQL replication including binlog formats, semi-sync, and group replication.
100- MongoDB Manual. ["Replication"](https://www.mongodb.com/docs/manual/replication/). MongoDB replica sets, automatic failover, and oplog-based replication.
101- Vogels, W. (2009). ["Eventually Consistent"](https://queue.acm.org/detail.cfm?id=1466448). *ACM Queue*. Practitioner essay on eventual consistency in replicated systems.
102- Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). ["A Comprehensive Study of Convergent and Commutative Replicated Data Types"](https://hal.inria.fr/inria-00555588). CRDT foundational paper; basis for multi-leader conflict resolution without data loss.
103- Brewer, E. (2012). ["CAP Twelve Years Later"](https://ieeexplore.ieee.org/document/6133253). *IEEE Computer*. Brewer's retrospective on CAP, useful for understanding the consistency-availability trade-offs replication realizes.
aibot88