# Replication Guide

> ReplicatedMergeTree operations, failover procedures, lag diagnosis, quorum writes, and Keeper management.

- Skill: `terrylica/replication-guide` (Agent Skill)
- Install (CLI): `npx skillmds@latest add terrylica/replication-guide`
- Raw SKILL.md: https://api.skillmd.com/api/skills/terrylica/replication-guide/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: terrylica (https://skillmd.com/u/terrylica)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/terrylica/replication-guide

---


# Replication Guide

## ReplicatedMergeTree Basics
- Engine: `ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')`
- Requires ZooKeeper or ClickHouse Keeper
- All replicas are equal — any replica can accept writes
- Replication is asynchronous by default

## Monitoring Replication
- `system.replicas` — per-table status: `absolute_delay`, `queue_size`, `is_leader`, `is_readonly`
- `system.replication_queue` — pending operations: fetches, merges, mutations
- Key health indicators:
  - `absolute_delay = 0` — fully caught up
  - `is_readonly = 0` — accepting writes
  - `queue_size < 10` — healthy queue
  - `active_replicas = total_replicas` — all replicas online

## Failover Procedures
1. Check replica status: `SELECT * FROM system.replicas WHERE is_readonly = 1`
2. Verify Keeper connectivity: `SELECT * FROM system.zookeeper WHERE path = '/'`
3. If replica is readonly due to Keeper disconnect, it auto-recovers when connection restores
4. For permanent failures: `SYSTEM DROP REPLICA 'replica_name' FROM TABLE db.table`
5. Corrupted metadata: `SYSTEM RESTORE REPLICA db.table`

## Quorum Writes
- `SET insert_quorum = 2` — wait for N replicas to confirm
- `SET insert_quorum_parallel = 1` — parallel quorum inserts (v21.8+)
- `SET insert_quorum_timeout = 60000` — timeout in ms
- Use for critical data that must survive node failures

## Keeper Management
- ClickHouse Keeper is the recommended replacement for ZooKeeper
- Monitor: `system.zookeeper` table for browsing ZK tree
- Key paths: `/clickhouse/tables/` for table metadata
- Check Keeper health: `SELECT * FROM system.asynchronous_metrics WHERE metric LIKE '%Keeper%'`

## Common Issues
- **Split brain**: Multiple leaders — usually Keeper issue, restart Keeper
- **Readonly replica**: Lost Keeper session — check network, Keeper logs
- **Queue buildup**: Slow fetches — check network bandwidth, disk I/O
- **Diverged replicas**: `SYSTEM SYNC REPLICA db.table` to force sync
- Load the `troubleshooting` skill for OOM-related replica failures and disk-full scenarios.