Database Replication Manager
Overview
Configure and manage database replication topologies for PostgreSQL (streaming replication, logical replication), MySQL (source-replica, group replication), and MongoDB (replica sets). This skill covers primary-replica setup, read scaling through replica routing, failover automation, replication lag monitoring, and conflict resolution for multi-primary configurations.
Prerequisites
- Superuser or replication-role credentials on primary and replica servers
- Network connectivity between all replication nodes (verify with
pg_isready or mysqladmin ping)
psql, mysql, or mongosh CLI tools installed on all nodes- Matching major database versions across all replication nodes
- Sufficient disk space on replicas (equal to or greater than primary)
- SSH access to replica servers for initial base backup transfer
Instructions
Choose the replication topology based on requirements:
- Single primary + read replicas: Best for read-heavy workloads. All writes go to primary; reads distributed across replicas.
- Multi-primary (active-active): Best for geographic distribution. Requires conflict resolution. Use PostgreSQL logical replication or MySQL Group Replication.
- Cascading replication: Replica A replicates from primary, Replica B replicates from Replica A. Reduces primary load for many replicas.
For PostgreSQL streaming replication, configure the primary:
- Set
wal_level = replica, max_wal_senders = 10, max_replication_slots = 10
- Create replication user:
CREATE ROLE replicator WITH REPLICATION LOGIN PASSWORD 'secure_password'
- Add replication entry to
pg_hba.conf: host replication replicator replica_ip/32 scram-sha-256
- Reload configuration:
SELECT pg_reload_conf()
Initialize the replica with a base backup: pg_basebackup -h primary_host -U replicator -D /var/lib/postgresql/data -Fp -Xs -P -R. The -R flag creates standby.signal and configures primary_conninfo automatically.
For MySQL source-replica replication, configure the source:
- Set
server-id = 1, log_bin = mysql-bin, binlog_format = ROW
- Create replication user:
CREATE USER 'replicator'@'replica_ip' IDENTIFIED BY 'secure_password'; GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'replica_ip'
- Record binary log position:
SHOW MASTER STATUS
Configure the MySQL replica: CHANGE REPLICATION SOURCE TO SOURCE_HOST='primary_host', SOURCE_USER='replicator', SOURCE_PASSWORD='...', SOURCE_LOG_FILE='mysql-bin.000001', SOURCE_LOG_POS=154; START REPLICA.
For MongoDB replica sets: initialize with rs.initiate({_id: "rs0", members: [{_id: 0, host: "node1:27017"}, {_id: 1, host: "node2:27017"}, {_id: 2, host: "node3:27017"}]}). MongoDB handles leader election and failover automatically.
Monitor replication lag continuously:
- PostgreSQL:
SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, (sent_lsn - replay_lsn) AS lag_bytes FROM pg_stat_replication
- MySQL:
SHOW REPLICA STATUS\G (check Seconds_Behind_Source)
- MongoDB:
rs.printReplicationInfo() and rs.printSecondaryReplicationInfo()
Configure application-level read routing: direct write queries to the primary connection and read queries to a load-balanced replica pool. Use connection poolers (PgBouncer, ProxySQL) or application middleware for automatic routing.
Set up automated failover using Patroni (PostgreSQL), MySQL InnoDB Cluster + MySQL Router, or MongoDB's built-in election mechanism. Test failover by stopping the primary and verifying the replica promotes automatically within the target RTO.
Configure alerting for replication lag exceeding 10 seconds, replication slot inactive for more than 1 hour, and replica connection drops. Stale replication slots in PostgreSQL cause WAL accumulation and can fill the disk.
Output
- Replication configuration files for primary and replica nodes
- Base backup and initialization scripts for setting up new replicas
- Replication monitoring queries with lag measurement and health checks
- Failover runbook with manual and automated promotion procedures
- Read routing configuration for application or connection pooler
Error Handling
| Error |
Cause |
Solution |
| Replication lag increasing steadily |
Replica cannot keep up with primary write volume |
Check replica I/O and CPU; increase max_parallel_workers on replica; consider upgrading replica hardware; reduce write-heavy batch operations on primary |
replication slot is inactive warning |
Replica disconnected or paused, WAL accumulating on primary |
Reconnect replica; if permanently removed, drop the slot with SELECT pg_drop_replication_slot('slot_name') to prevent disk fill |
could not connect to primary on replica |
Network partition, primary down, or authentication failure |
Verify network connectivity; check pg_hba.conf entries; confirm replication user credentials; check primary process status |
| Replica has diverged from primary |
Split-brain after failed failover or manual writes to replica |
Re-initialize replica from fresh base backup; for PostgreSQL, use pg_rewind if timeline divergence is small |
| Conflict in logical replication |
Same row modified on both publisher and subscriber |
Configure conflict resolution policy; use UPDATE conflict handler; design schema to avoid cross-node writes to same rows |
Examples
Setting up PostgreSQL read replicas for a web application: A primary database handles 2,000 writes/second but read traffic is 10x higher. Two streaming replicas are added with PgBouncer routing read queries to replicas in round-robin. Result: primary CPU drops from 90% to 40%, read latency improves by 60%.
Automated failover with Patroni: A 3-node PostgreSQL cluster managed by Patroni with etcd for consensus. When the primary fails, Patroni promotes the most up-to-date replica within 10 seconds. Application reconnects automatically through the Patroni-managed VIP or DNS endpoint.
Cross-region logical replication for compliance: EU customer data must stay in EU region. Logical replication publishes only non-PII tables to the US region replica. EU application reads locally; US analytics queries use the replicated subset. Publication filter: CREATE PUBLICATION us_analytics FOR TABLE orders, products, categories.
Resources
1---2name: managing-database-replication3description: Process use when you need to work with database scalability. This skill provides replication and sharding with comprehensive guidance and automation. Trigger with phrases like "set up replication", "implement sharding", or "scale database".4license: MIT5---6# Database Replication Manager
7
8## Overview
9
10Configure and manage database replication topologies for PostgreSQL (streaming replication, logical replication), MySQL (source-replica, group replication), and MongoDB (replica sets). This skill covers primary-replica setup, read scaling through replica routing, failover automation, replication lag monitoring, and conflict resolution for multi-primary configurations.
11
12## Prerequisites
13
14- Superuser or replication-role credentials on primary and replica servers
15- Network connectivity between all replication nodes (verify with `pg_isready` or `mysqladmin ping`)
16- `psql`, `mysql`, or `mongosh` CLI tools installed on all nodes
17- Matching major database versions across all replication nodes
18- Sufficient disk space on replicas (equal to or greater than primary)
19- SSH access to replica servers for initial base backup transfer
20
21## Instructions
22
231. Choose the replication topology based on requirements:
24 - **Single primary + read replicas**: Best for read-heavy workloads. All writes go to primary; reads distributed across replicas.
25 - **Multi-primary (active-active)**: Best for geographic distribution. Requires conflict resolution. Use PostgreSQL logical replication or MySQL Group Replication.
26 - **Cascading replication**: Replica A replicates from primary, Replica B replicates from Replica A. Reduces primary load for many replicas.
27
282. For PostgreSQL streaming replication, configure the primary:
29 - Set `wal_level = replica`, `max_wal_senders = 10`, `max_replication_slots = 10`
30 - Create replication user: `CREATE ROLE replicator WITH REPLICATION LOGIN PASSWORD 'secure_password'`
31 - Add replication entry to `pg_hba.conf`: `host replication replicator replica_ip/32 scram-sha-256`
32 - Reload configuration: `SELECT pg_reload_conf()`
33
343. Initialize the replica with a base backup: `pg_basebackup -h primary_host -U replicator -D /var/lib/postgresql/data -Fp -Xs -P -R`. The `-R` flag creates `standby.signal` and configures `primary_conninfo` automatically.
35
364. For MySQL source-replica replication, configure the source:
37 - Set `server-id = 1`, `log_bin = mysql-bin`, `binlog_format = ROW`
38 - Create replication user: `CREATE USER 'replicator'@'replica_ip' IDENTIFIED BY 'secure_password'; GRANT REPLICATION SLAVE ON *.* TO 'replicator'@'replica_ip'`
39 - Record binary log position: `SHOW MASTER STATUS`
40
415. Configure the MySQL replica: `CHANGE REPLICATION SOURCE TO SOURCE_HOST='primary_host', SOURCE_USER='replicator', SOURCE_PASSWORD='...', SOURCE_LOG_FILE='mysql-bin.000001', SOURCE_LOG_POS=154; START REPLICA`.
42
436. For MongoDB replica sets: initialize with `rs.initiate({_id: "rs0", members: [{_id: 0, host: "node1:27017"}, {_id: 1, host: "node2:27017"}, {_id: 2, host: "node3:27017"}]})`. MongoDB handles leader election and failover automatically.
44
457. Monitor replication lag continuously:
46 - PostgreSQL: `SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, (sent_lsn - replay_lsn) AS lag_bytes FROM pg_stat_replication`
47 - MySQL: `SHOW REPLICA STATUS\G` (check `Seconds_Behind_Source`)
48 - MongoDB: `rs.printReplicationInfo()` and `rs.printSecondaryReplicationInfo()`
49
508. Configure application-level read routing: direct write queries to the primary connection and read queries to a load-balanced replica pool. Use connection poolers (PgBouncer, ProxySQL) or application middleware for automatic routing.
51
529. Set up automated failover using Patroni (PostgreSQL), MySQL InnoDB Cluster + MySQL Router, or MongoDB's built-in election mechanism. Test failover by stopping the primary and verifying the replica promotes automatically within the target RTO.
53
5410. Configure alerting for replication lag exceeding 10 seconds, replication slot inactive for more than 1 hour, and replica connection drops. Stale replication slots in PostgreSQL cause WAL accumulation and can fill the disk.
55
56## Output
57
58- **Replication configuration files** for primary and replica nodes
59- **Base backup and initialization scripts** for setting up new replicas
60- **Replication monitoring queries** with lag measurement and health checks
61- **Failover runbook** with manual and automated promotion procedures
62- **Read routing configuration** for application or connection pooler
63
64## Error Handling
65
66| Error | Cause | Solution |
67|-------|-------|---------|
68| Replication lag increasing steadily | Replica cannot keep up with primary write volume | Check replica I/O and CPU; increase `max_parallel_workers` on replica; consider upgrading replica hardware; reduce write-heavy batch operations on primary |
69| `replication slot is inactive` warning | Replica disconnected or paused, WAL accumulating on primary | Reconnect replica; if permanently removed, drop the slot with `SELECT pg_drop_replication_slot('slot_name')` to prevent disk fill |
70| `could not connect to primary` on replica | Network partition, primary down, or authentication failure | Verify network connectivity; check `pg_hba.conf` entries; confirm replication user credentials; check primary process status |
71| Replica has diverged from primary | Split-brain after failed failover or manual writes to replica | Re-initialize replica from fresh base backup; for PostgreSQL, use `pg_rewind` if timeline divergence is small |
72| Conflict in logical replication | Same row modified on both publisher and subscriber | Configure conflict resolution policy; use `UPDATE` conflict handler; design schema to avoid cross-node writes to same rows |
73
74## Examples
75
76**Setting up PostgreSQL read replicas for a web application**: A primary database handles 2,000 writes/second but read traffic is 10x higher. Two streaming replicas are added with PgBouncer routing read queries to replicas in round-robin. Result: primary CPU drops from 90% to 40%, read latency improves by 60%.
77
78**Automated failover with Patroni**: A 3-node PostgreSQL cluster managed by Patroni with etcd for consensus. When the primary fails, Patroni promotes the most up-to-date replica within 10 seconds. Application reconnects automatically through the Patroni-managed VIP or DNS endpoint.
79
80**Cross-region logical replication for compliance**: EU customer data must stay in EU region. Logical replication publishes only non-PII tables to the US region replica. EU application reads locally; US analytics queries use the replicated subset. Publication filter: `CREATE PUBLICATION us_analytics FOR TABLE orders, products, categories`.
81
82## Resources
83
84- PostgreSQL streaming replication: https://www.postgresql.org/docs/current/warm-standby.html
85- PostgreSQL logical replication: https://www.postgresql.org/docs/current/logical-replication.html
86- MySQL replication: https://dev.mysql.com/doc/refman/8.0/en/replication.html
87- Patroni (PostgreSQL HA): https://patroni.readthedocs.io/
88- MongoDB replica sets: https://www.mongodb.com/docs/manual/replication/
micsapp