Mongodb Replica Sets

MongoDB Replica Sets in Production

How to run a replica set, not how to query it (that's mongodb-rules). These are the operational decisions Claude tends to get wrong.

Topology: odd voting members, real hostnames, never localhost

Use an odd number of voting members, three, so a majority can still elect a primary when one is lost. A two-member set has no majority if either dies, it goes read-only. Avoid arbiters unless you truly must: an arbiter holds no data, so a three-node primary-secondary-arbiter set that loses its one data secondary can no longer satisfy w:"majority". Address every member by a hostname that all other members and all clients can resolve and reach; localhost in the config is a classic break, the other members can't reach it. Initialize once, from a single member, after all members are up, and wait for the election before creating users.

rs.initiate({ _id: "rs0", members: [
  { _id: 0, host: "mongo1.internal:27017", priority: 2 },
  { _id: 1, host: "mongo2.internal:27017", priority: 1 },
  { _id: 2, host: "mongo3.internal:27017", priority: 1 }
]})

Connect to the set, not to one node

The connection string must list the seed members and name the set, so the driver can find the current primary and follow failover. Pointing the app at a single host throws away the high availability the replica set exists to provide.

mongodb://mongo1,mongo2,mongo3/db?replicaSet=rs0

Durability and reads

Writes always go to the primary. w:"majority" means a majority of members acknowledged before the app gets the OK, that is durability, use it for data you can't lose; w:1 (primary only) is fine for the regenerable. Add wtimeoutMS so a degraded set doesn't block writes forever. Reads default to the primary and are strongly consistent. Secondaries are eventually consistent, they lag, so only route reads to them (secondaryPreferred, secondary, or nearest) when stale data is acceptable, analytics and reporting, not a read-after-write the user just made.

Authentication between members

A replica set needs internal auth or any host can join. Generate a keyfile (openssl rand -base64 756), share it across members, and enable security.authorization: enabled with the keyfile. Never expose 27017 to the internet, bind to the private network and firewall it. Use TLS for client and inter-node traffic when the network isn't fully trusted.

OS tuning MongoDB actually requires (self-hosted)

These aren't optional polish, MongoDB warns about them and they cause real instability if skipped:

• Disable Transparent Huge Pages (THP). THP hurts database memory access patterns badly; set enabled and defrag to never at boot.
• vm.swappiness=1. Keep the working set in RAM instead of swapping out the cache.
• Raise ulimits. nofile and nproc to 64000/32000, the defaults are too low for a busy mongod's connections and threads.
• XFS for the data volume. MongoDB recommends XFS over ext4 for WiredTiger. Never put data on NFS or network storage, the latency wrecks it.

WiredTiger cache, especially in a container

Set --wiredTigerCacheSizeGB explicitly from the container's memory limit, roughly half of it minus 1GB, and cap the container's memory in the deploy block. Don't rely on the default: it targets about half of system RAM, and the cache is only part of mongod's footprint (connections, aggregation, and sort buffers live outside it). An unset cache sized against the host plus those buffers will exceed a container limit and get the container OOM-killed. Leave headroom on purpose.

Running it in Docker or Swarm

The replica-set-specific traps on top of the general docker and docker-swarm skills:

• Publish the port in mode: host, never ingress. The routing mesh load-balances 27017 across all three members, which breaks the replica set, clients and members must reach a specific member. Use ports: [{ target: 27017, published: 27017, mode: host }].
• Pin each member to its own node. Without placement constraints all three can land on one host and you have zero fault tolerance. Label nodes and constrain each service (node.labels.mongo.replica == 1), one member per host.
• Bind-mount the data directory to a host path. Anonymous or named-without-bind volumes risk data loss on recreation and are hard to back up. Pre-create the dir and chown 999:999 (the image runs as UID 999). XFS host filesystem.
• Real healthcheck, with a start period. test: ["CMD","mongosh","--eval","db.adminCommand('ping')"] and start_period: 60s, so a slow first start isn't read as unhealthy.
• Keyfile and root password via Docker secrets, not plaintext env (which shows in docker inspect and image history).

Oplog and failover discipline

The oplog is your recovery window: a secondary that falls further behind than the oplog covers needs a full resync, and point-in-time recovery can only reach back as far as the oplog. Size it for your write volume (24h minimum, 48 to 72h is a safer target; db.getReplicationInfo() shows the current window). Test failover on a schedule with rs.stepDown(), don't discover at an outage that the app doesn't reconnect. Roll upgrades secondaries-first, one at a time, step the primary down last, and let each node resync before moving on.

This skill is built to grow. Add a rule when a real replica-set operation has a stable, defensible fix.