Enterprise High Availability

Enterprise High Availability

Purpose

Design and operate systems that hold availability targets from 95% to 99.999% across the full stack: database replica + slave topology, load balancer health contracts, zero-downtime application version rollout, schema/data sync during migration, and quorum-based failover. Applies equally to a solo-built enterprise SaaS and a 10k-node Fortune-500 platform — only the budget, redundancy factor, and automation depth change.

Agent Protocol

Trigger

Exact user phrases: "high availability", "HA", "99.9", "99.99", "99.999", "five nines", "four nines", "three nines", "uptime target", "replica", "read replica", "slave db", "standby", "load balancer", "LB", "HAProxy", "Nginx LB", "ALB", "NLB", "Envoy", "blue-green", "canary", "rolling update", "zero downtime", "version sync", "schema migration", "data sync", "backfill", "dual write", "expand contract", "failover", "quorum", "split brain", "RPO", "RTO", "active-active", "active-passive", "multi-AZ", "multi-region".

Input Context

Before activating, verify:

• Target availability tier (95 / 99 / 99.9 / 99.99 / 99.999) and the SLA penalty if missed
• Current architecture: single node? master-slave? multi-AZ? multi-region?
• Database engine + replication mode (sync, semi-sync, async)
• Load balancer layer (L4 TCP, L7 HTTP) and current health-check semantics
• Deployment model (bare metal, VM, Kubernetes, serverless)
• Migration cadence (per-day / per-week) and current downtime per release
• Read/write ratio, peak TPS, dataset size, max acceptable replication lag
• Team size and on-call capacity (drives automation vs manual runbook split)
• Budget ceiling — five nines costs 10–100× more than three nines

Output Artifact

HA architecture spec containing: availability tier + error budget, replica topology, LB config with health-check contract, version rollout strategy, migration plan with data sync, failover runbook, and monitoring/alerting matrix.

Response Format

Tier: {99 | 99.9 | 99.99 | 99.999}
Error budget: {minutes/year, minutes/month}
Topology: {single-AZ | multi-AZ active-passive | multi-AZ active-active | multi-region active-active}
DB: {primary + N replicas, sync mode, RPO, RTO}
LB: {L4/L7, algorithm, health-check, drain time}
Rollout: {blue-green | canary | rolling}, batch size, bake time
Migration: {expand-contract phases, dual-write window, backfill plan}
Failover: {trigger, quorum, promotion, DNS/anycast cutover, rollback}

# Concrete config snippets for LB, replication, deployment

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.

Completion Criteria

• Availability tier selected with error-budget minutes computed
• Replica topology defined with sync mode and replica count per AZ/region
• Load balancer config with health-check path, interval, threshold, drain
• Version rollout strategy (blue-green / canary / rolling) with bake + rollback time
• Schema migration plan using expand-contract (N+1 compatible)
• Data sync plan: dual-write window, backfill job, cutover step, verify step
• Quorum and split-brain prevention mechanism documented
• Failover runbook with RPO/RTO targets and rollback path
• Monitoring: replication lag, LB pool health, error budget burn, deploy success
• DR drill cadence scheduled (quarterly minimum for ≥99.99%)

Max Response Length

350 lines of code and configuration.

Workflow

Step 1: Pick Availability Tier and Compute Budget

Availability is a cost decision, not a vanity number. Pick the lowest tier that satisfies the SLA + revenue impact.

Tier      Yearly downtime   Monthly downtime   Typical use case
95%       18d 6h            36h 30m            Internal tools, dev/test
99%       3d 15h            7h 18m             Beta SaaS, B2C non-critical
99.9%     8h 45m            43m 12s            Standard SaaS, ecommerce
99.95%    4h 22m            21m 36s            Premium SaaS, B2B
99.99%    52m 35s           4m 19s             Fintech, healthcare, ads
99.999%   5m 15s            25.9s              Telecom, trading, life-safety
99.9999%  31.5s             2.59s              Tier-1 telco core, exchanges

Rule of thumb: each extra nine multiplies cost by ~5–10×. >99.99% requires multi-region active-active, automated failover, anycast/GeoDNS, and full DR drills.

Step 2: Choose Replica Topology

Single AZ + single node              → max 99.0%   (one EC2/PG box, daily snapshot)
Single AZ + primary + 1 sync replica → max 99.5%   (semi-sync, manual promote)
Multi-AZ active-passive              → max 99.95%  (primary + standby in AZ-b, auto promote)
Multi-AZ active-active (read)        → max 99.99%  (writes to primary, reads load-balanced across AZ replicas)
Multi-region active-passive          → max 99.99%  (DR region with async replica, GeoDNS failover)
Multi-region active-active           → 99.999%+    (multi-master / global DB: Spanner, Aurora Global, CockroachDB)

Replica count formula for read scaling: replicas = ceil(read_tps / per_replica_tps) + 1 (spare). Cap at 5 per primary — beyond that, fan out via cascading or use a sharded primary.

Step 3: Slave/Standby DB Configuration

Always use semi-sync for same-AZ standby (ack from ≥1 replica before commit), async for cross-region. GTID/LSN-based replication so promotion is deterministic.

# MySQL primary (my.cnf) — semi-sync, GTID, row-based
[mysqld]
server_id = 1
log_bin = mysql-bin
binlog_format = ROW
gtid_mode = ON
enforce_gtid_consistency = ON
sync_binlog = 1
innodb_flush_log_at_trx_commit = 1
plugin_load_add = semisync_master.so
rpl_semi_sync_master_enabled = 1
rpl_semi_sync_master_timeout = 3000        # fallback to async after 3s
rpl_semi_sync_master_wait_for_slave_count = 1
rpl_semi_sync_master_wait_point = AFTER_SYNC

# Replica
[mysqld]
server_id = 2
read_only = ON
super_read_only = ON
gtid_mode = ON
enforce_gtid_consistency = ON
relay_log_recovery = ON
plugin_load_add = semisync_slave.so
rpl_semi_sync_slave_enabled = 1

# PostgreSQL primary — streaming + slot, synchronous_commit per workload
wal_level = replica
max_wal_senders = 10
max_replication_slots = 10
synchronous_standby_names = 'ANY 1 (standby_az_b, standby_az_c)'
synchronous_commit = on            # use 'remote_apply' only for RPO=0 critical writes

Step 4: Load Balancer in Front of App + DB Reads

L4 (TCP) for raw throughput, L7 (HTTP) for path routing, header-based canary, and rich health checks. Active health check + passive ejection. Connection draining required for zero-downtime deploys.

# HAProxy — L7 with active+passive health, drain, retries
global
  maxconn 50000
  log stdout local0
defaults
  mode http
  timeout connect 2s
  timeout client 30s
  timeout server 30s
  option httplog
  retries 2
  option redispatch

frontend fe_app
  bind *:443 ssl crt /etc/ssl/app.pem alpn h2,http/1.1
  default_backend be_app

backend be_app
  balance leastconn
  option httpchk GET /healthz
  http-check expect status 200
  default-server inter 2s fall 3 rise 2 slowstart 30s observe layer7 error-limit 10 on-error mark-down
  server app-1 10.0.1.10:8080 check
  server app-2 10.0.1.11:8080 check
  server app-3 10.0.2.10:8080 check
  # drain via: socat - /var/run/haproxy.sock <<< "set server be_app/app-1 state drain"

backend be_db_reads
  balance roundrobin
  option mysql-check user haproxy_check
  server replica-1 10.0.1.20:3306 check
  server replica-2 10.0.2.20:3306 check
  server replica-3 10.0.3.20:3306 check backup   # last resort

Health-check contract MUST be defined by the app team:

• /healthz = process alive (liveness, fast, no deps)
• /readyz = ready to serve (DB connected, cache warm, deps reachable)
• LB hits /readyz; orchestrator hits /healthz

Step 5: Zero-Downtime App Version Rollout

Choose by error-budget pressure:

Rolling      → cheapest, slowest, partial blast radius. Default for stateless apps.
Blue-green   → instant cutover + instant rollback, 2× infra cost during deploy.
Canary       → safest, gradual % shift (1 → 5 → 25 → 50 → 100), needs metric automation.
Shadow       → mirror prod traffic to new version, compare results, zero user impact.

# Kubernetes rolling — surge 25% / unavailable 0, preStop drain
apiVersion: apps/v1
kind: Deployment
metadata: {name: api}
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate: {maxSurge: 25%, maxUnavailable: 0}
  minReadySeconds: 20
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: api
        image: api:v2.4.0
        readinessProbe:
          httpGet: {path: /readyz, port: 8080}
          periodSeconds: 2
          failureThreshold: 3
        livenessProbe:
          httpGet: {path: /healthz, port: 8080}
          periodSeconds: 10
        lifecycle:
          preStop:
            exec: {command: ["/bin/sh","-c","sleep 15 && curl -X POST localhost:8080/drain"]}

# Argo Rollouts canary with analysis gate
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 5m}
      - analysis: {templates: [{templateName: success-rate}]}
      - setWeight: 25
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 10m}
      - setWeight: 100

Step 6: Version Sync Across Nodes (N and N+1 Must Coexist)

During rolling deploy, both v1 and v2 of the app run simultaneously and hit the same DB. The system MUST be N/N+1 compatible:

• API: add new fields, never remove or rename in same release (deprecate over ≥2 releases)
• Wire format: new fields optional, old clients ignore unknown fields (protobuf, JSON)
• Feature flags: ship code dark, flip atomically after 100% rollout
• Config: new keys default to old behavior; remove old keys one release later
• Background jobs: queue payload versioned; consumers handle v1 and v2 simultaneously

Release N    Release N+1   Release N+2
add column   read+write    drop old column
write both   only new      ---

Pin compatibility matrix in CI: deploy v1 + v2 in same cluster, run smoke + contract tests against both.

Step 7: Schema Migration — Expand / Contract

Never break running pods. Multi-phase migration over multiple releases.

Phase 1 EXPAND  (release N)
  - Add new column (nullable) / new table / new index CONCURRENTLY
  - App still reads + writes OLD shape
  - SAFE: zero downtime, fully reversible

Phase 2 DUAL WRITE  (release N)
  - App writes to BOTH old and new shape
  - App reads from OLD (source of truth)
  - Backfill historical rows into new shape (batched, throttled)

Phase 3 CUTOVER  (release N+1)
  - App reads from NEW shape (with fallback to OLD)
  - Verify row counts + checksums
  - Flip feature flag atomically

Phase 4 CONTRACT  (release N+2, after bake)
  - Stop writing to OLD shape
  - Drop OLD column / table (after retention window for rollback)

-- Phase 1: PostgreSQL safe DDL
ALTER TABLE orders ADD COLUMN customer_uuid uuid NULL;            -- instant
CREATE INDEX CONCURRENTLY idx_orders_customer_uuid ON orders(customer_uuid);
-- DO NOT: ADD COLUMN ... NOT NULL DEFAULT  (rewrites table on old PG)
-- DO NOT: rename in place — add new, dual-write, drop old

-- Backfill in throttled batches, idempotent
DO $$
DECLARE batch_size int := 10000; rows int;
BEGIN
  LOOP
    UPDATE orders o SET customer_uuid = c.uuid
    FROM customers c WHERE o.customer_id = c.id AND o.customer_uuid IS NULL
    AND o.id IN (SELECT id FROM orders WHERE customer_uuid IS NULL LIMIT batch_size);
    GET DIAGNOSTICS rows = ROW_COUNT;
    EXIT WHEN rows = 0;
    PERFORM pg_sleep(0.1);
  END LOOP;
END $$;

Step 8: Data Sync During Migration (Dual Write + Verify)

Use the outbox pattern + idempotent consumer, or transactional dual-write with reconciliation job.

App write
   |
   +-- INSERT INTO orders ...
   +-- INSERT INTO outbox (event_id, payload, status='pending')   -- same tx
   |
   v
Outbox relay (poll or Debezium) -> Kafka topic -> Consumer -> NEW store
                                                    |
                                                    +--> reconciliation: every 5m
                                                         compare row counts + checksum
                                                         alert if drift > 0.01%

Verification gates BEFORE cutover:

• Row count parity (per shard / per tenant)
• Checksum of canonical fields (xxhash64 of normalized JSON)
• Sample 1k recent rows, compare deep-equal
• Replay window: last 24h must match exactly

Step 9: Quorum + Split-Brain Prevention

Any auto-failover needs an odd-numbered consensus group. Never 2 — it cannot survive a partition without risk of split-brain (both sides think they are primary).

Group size   Failures tolerated   Notes
1            0                    no HA
3            1                    minimum viable
5            2                    standard prod
7            3                    geo-distributed

Use etcd / Consul / ZooKeeper / Raft built into the DB (CockroachDB, TiDB, MongoDB). Lease-based leader. Fencing token on every primary action so a zombie old-primary cannot commit.

Step 10: Failover Runbook (RPO/RTO Targets)

Tier      RPO target   RTO target   Failover mode
99.0%     1h           4h           Backup restore
99.9%     5m           30m          Async replica, semi-auto promote
99.95%    1m           5m           Semi-sync replica, auto promote in AZ
99.99%    <10s         <60s         Sync replica + LB pool swap, auto + GeoDNS
99.999%   0            <10s         Multi-master, anycast, no human in loop

Runbook skeleton (must be tested quarterly for ≥99.99%):

1. DETECT     — alert on primary unreachable for >N seconds (N = RTO/3)
2. FENCE      — revoke old primary's lease token; block its writes
3. PROMOTE    — quorum elects new primary (or manual cmd: pg_ctl promote / FAILOVER)
4. REROUTE    — update LB pool / VIP / DNS / Consul service; TTL ≤ RTO/2
5. VALIDATE   — health check + canary read + canary write succeed
6. NOTIFY     — page on-call, post status page, log incident
7. REBUILD    — old primary becomes new standby (pg_rewind / reseed)
8. POSTMORTEM — within 48h; update runbook; adjust budget

Step 11: Monitoring + Burn-Rate Alerting

Required signals (alert before users notice):

• Replication lag: warn @ 5s, page @ 30s (async); warn @ 1s for semi-sync
• LB pool health: page if healthy < 50%
• App error rate: multi-window burn (2% budget in 1h → page; 5% in 6h → page)
• Deploy success rate: page if last 3 deploys failed
• DB connection pool saturation: warn @ 80%, page @ 95%
• Disk: warn @ 75%, page @ 85% (with 4h growth projection)

Rules

• Pick the lowest availability tier the SLA allows; every extra nine costs 5–10× more.
• Two-node clusters are forbidden for auto-failover (use 3 or 5).
• Semi-sync for same-AZ, async for cross-region; sync only for RPO=0 financial writes.
• LB must use /readyz (deep) for pool membership, /healthz (shallow) for orchestrator.
• App releases MUST be backward + forward compatible across one release boundary.
• Schema migrations MUST follow expand-contract; no in-place rename / drop in same release.
• Every destructive DDL has a bake window ≥ 1 release before contract phase.
• Connection draining ≥ 15s, terminationGracePeriod ≥ 60s on Kubernetes.
• All auto-failover paths drilled quarterly with real traffic for ≥99.99%.
• Burn-rate alerts use multi-window (fast + slow burn), never single-threshold.
• Never deploy on Friday afternoon to a 99.99% system without senior approval.

Architecture Decision Trees

Rollout Strategy Selection

Error budget available?
├── Generous (> 50% remaining) → Rolling update
│   Cheapest. 25% surge, 0 unavailable. Bake time 20s.
│   Risk: partial blast radius, slow rollback (per pod).
├── Moderate (25-50%) → Canary
│   5% → 25% → 50% → 100% with analysis gates.
│   Automated rollback if error rate spikes.
└── Tight (< 25%) → Blue-green
    Instant cutover. Instant rollback. 2x infra cost.
    Best for: zero-downtime mandatory, critical compliance.

Replication Mode Selection

RPO requirement?
├── Zero data loss → Synchronous replication
│   Commit waits for ≥1 replica ack. Increases p99 latency by 1-5ms.
│   Use only for financial transactions. Budget for latency impact.
├── < 5s data loss → Semi-synchronous
│   Default for same-AZ. Timeout falls back to async.
│   RPO = 0 during normal ops, < 1s during degredation.
└── < 60s data loss → Asynchronous
    Cross-region only. No latency impact on primary.
    Risk: replication lag causes data loss on failover.

Implementation Patterns

Pattern: Blue-Green Deployment with LB Swap

# argo-rollouts/bluegreen.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: api-service}
spec:
  replicas: 10
  strategy:
    blueGreen:
      activeService: api-active
      previewService: api-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:
        templates:
        - templateName: smoke-test
      scaleDownDelaySeconds: 300  # keep old version for rollback
  template:
    spec:
      containers:
      - name: api
        image: api:v2.5.0
        readinessProbe:
          httpGet: {path: /readyz, port: 8080}
          periodSeconds: 2
          failureThreshold: 2
        lifecycle:
          preStop:
            exec: {command: ["/bin/sh","-c","sleep 15"]}
---
apiVersion: v1
kind: Service
metadata: {name: api-active}
spec:
  selector: {app: api, rollouts-pod-template-hash: "stable"}

Pattern: Semi-Sync Replication with Auto-Failover

# Patroni configuration for PostgreSQL HA
scope: mydb
namespace: /service/
name: pg-primary

restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.0.1.10:8008

etcd:
  hosts: ['10.0.1.100:2379', '10.0.1.101:2379', '10.0.1.102:2379']

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576  # 1MB
    postgresql:
      use_pg_rewind: true
      parameters:
        wal_level: replica
        hot_standby: "on"
        wal_log_hints: "on"
        synchronous_standby_names: "ANY 1 (pg-replica-1, pg-replica-2)"

postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.0.1.10:5432
  data_dir: /data/postgresql
  bin_dir: /usr/lib/postgresql/16/bin
  authentication:
    replication:
      username: replicator
      password: strongpassword
  parameters:
    max_connections: 200

Production Considerations

Disaster Recovery Drills

• Quarterly for ≥99.99%: scheduled failover exercise during maintenance window. Half production traffic routed to DR.
• Annual: full region failover. All traffic to DR region for 4 hours. Measure RPO and RTO.
• Game day: surprise failover scenario. Team follows runbook without prior notice. Measure response time.
• Post-drill: update runbook based on findings. Add automation for manual steps discovered during drill.

Cost Management

• Multi-AZ DB: 2x compute cost for standby. ~20% increase for storage (replication).
• Blue-green deploy: 2x infra during deploy window. Use scaled-down preview environment for cost savings.
• Cross-region: minimum 2x infra cost. Data transfer costs between regions.
• TURN relay: $0.005-0.02 per GB egress. Budget 2-5 Mbps per active media stream.

Anti-Patterns

Performance Optimization

• Connection pooling with PgBouncer: reduce PostgreSQL connection overhead. Transaction pooling mode.
• Read replicas for reporting: offload analytics queries from primary. Allow 5-10 replicas.
• Database connection limit: 100 per application instance. Queue with pgbouncer for spikes.
• Caching layer at LB: cache GET responses for 30s. Reduces application load by 30-50%.
• Query optimization: slow query log (>200ms). Index recommendations from pg_stat_statements.
• Replica lag monitoring: alert on > 5s lag for async, > 1s for semi-sync. Investigate immediately.
• Disk throughput: provision IOPS at 3x baseline. Burst credits for gp3 volumes monitored.

Security Considerations

• TLS everywhere: mTLS between services. Certificate rotation every 90 days. Auto-renew via cert-manager.
• Database encryption at rest: AWS RDS encryption / Azure TDE. Key rotation every 12 months.
• Network segmentation: app and DB in private subnets. Bastion host for admin access.
• Backup encryption: S3 bucket with SSE-KMS. Cross-region backup copy encrypted with different key.
• Access control: database credentials in Vault. Rotated every 30 days. Application reads at startup.
• Audit logging: all schema changes logged. DDL triggers in PostgreSQL. Review weekly.
• Failover authentication: failover commands require MFA. Human-in-the-loop for manual promotion.
• WAF in front of LB: rate limiting, SQL injection protection, IP blocklist for known bad actors.

Handoff

• enterprise-sla-management for SLA contract wording, customer credits, multi-tier SLA structure.
• data-data-replication for deep dive on database-specific replication internals (GoldenGate, Patroni, etc).
• data-cdc-patterns when dual-write is implemented via Debezium / Kafka Connect.
• devops-progressive-delivery for Argo Rollouts / Flagger canary automation specifics.
• devops-incident-response for on-call paging, escalation, status page automation.
• devops-backup-dr for backup cadence, snapshot retention, restore drills.