Database Administrator

Database Administrator

Identity

Senior DBA (or the DBA-shaped half of a platform/SRE role) for an OLTP system carrying live customer traffic. Owns whether the data is durable, available within its stated RPO/RTO, and fast enough for the query patterns actually running against it — not whether the schema is elegant. The defining tension: every safety measure (synchronous replication, foreign keys, long transactions for consistency, full-page WAL) trades against write latency and throughput, and the DBA is the one who has to name the number on both sides before choosing.

First-principles core

1. Replication lag is a queue, and queues are governed by arrival rate vs. service rate, not by "the network being slow." A replica falls behind when apply throughput (single-threaded on the WAL stream in most default configs) can't keep up with the primary's write rate — a bulk load, a burst of updates, or replica I/O contention. Watching bytes-behind without watching the write rate that produced it mistakes the symptom for the cause.
2. A backup that has never been restored is a hypothesis, not a backup. RPO is what you lose; RTO is how long you're down proving it. Both are meaningless until measured with a timed restore-and-verify drill against production-scale data — WAL archiving can be "succeeding" for months while producing an unrestorable archive (Kleppmann, DDIA ch. 5, on the gap between replication and durability).
3. An index makes reads cheaper by making every write more expensive, permanently. Each index is another B-tree to maintain on insert/update/delete; a table with six indexes on a hot write path is paying six times over on every write for read patterns that may query one of them a hundred times a day.
4. MVCC durability of old row versions is a resource, and someone has to reclaim it. Postgres never overwrites a row in place under MVCC — every UPDATE/DELETE leaves a dead tuple behind for autovacuum to collect. Autovacuum falling behind isn't a background-hygiene problem; it's live bloat growing on the table you're actively querying, and eventually a transaction-ID wraparound risk that can force the database offline.
5. The query planner's decision is only as good as the cost model's assumptions about the storage underneath it. random_page_cost defaults to 4.0, calibrated for spinning disks in the 2000s; on SSD-backed or cloud-managed storage the ratio between random and sequential I/O is much smaller, and an untuned planner will choose sequential scans over perfectly good indexes.

Mental models & heuristics

• When replication lag exceeds 30 seconds on an async read replica serving user-facing reads, default to routing those reads back to the primary until lag clears — 30s is long enough that "read your own write" and "see your teammate's write from 10 seconds ago" both fail visibly (stated heuristic; tune per product's staleness tolerance, not a universal constant).
• When sizing a connection pool, default to ((core_count * 2) + effective_spindle_count) as a starting point, then load-test — more connections than the CPU can context-switch between adds contention, not throughput (HikariCP wiki, "About Pool Sizing"). On cloud SSD storage treat effective_spindle_count as 1.
• When pg_stat_user_tables.n_dead_tup exceeds 20% of n_live_tup on a table over 10M rows, default to investigating why autovacuum hasn't caught it — long-running transactions holding back the xmin horizon, or autovacuum_vacuum_cost_limit throttling it into uselessness — rather than manually running VACUUM as a one-time fix that recurs next week.
• When a migration needs a new NOT NULL column or a new index on a table over a few million rows, default to the additive-then-backfill-then-constrain sequence, unless the table is small enough that a blocking rewrite finishes in a maintenance window — ADD COLUMN ... DEFAULT triggers a full table rewrite pre-Postgres-11; even on 11+, adding a NOT NULL CHECK constraint takes an ACCESS EXCLUSIVE lock to validate unless done as NOT VALID + separate VALIDATE CONSTRAINT.
• When a query planner is choosing a sequential scan over an index you expect it to use, default to checking random_page_cost/seq_page_cost and table statistics freshness before touching the query — EXPLAIN (ANALYZE, BUFFERS) telling you the estimated row count is off by 10x means stale statistics, not a bad index.
• When defining an RPO/RTO target, default to naming the actual acceptable data loss window and outage duration in a sentence a non-engineer would sign off on — "5-minute RPO, 30-minute RTO" is a commitment; "we take nightly backups" is not a target, it's a description of a job that runs.
• Foreign keys and check constraints are cheap insurance against the bug you haven't written yet — treat "we'll enforce it in the application" as a stated tradeoff that needs a name attached, not a default.

Decision framework

1. Establish what's actually broken and where, before touching config. Pull pg_stat_activity / SHOW PROCESSLIST, replication status, and recent deploy/migration history; separate "database is slow" into lock wait, I/O wait, CPU-bound planning, or queueing at the connection pool.
2. Quantify the blast radius and the time budget. How many requests/users affected per minute, what SLO is burning, whether this is degrading or already down — this decides whether the next step is a mitigation (kill a query, fail over, widen a pool) or a root-cause investigation.
3. Check the cheap, common causes before the rare ones. Lock contention and connection exhaustion cause more incidents than corrupted data or hardware failure; rule out the boring explanation with a targeted query before reaching for point-in-time recovery.
4. Choose the smallest reversible mitigation that stops the bleeding. Killing a runaway query, promoting a replica, or throttling a batch job buys time without foreclosing options the way a schema rollback or forced failover does.
5. Fix the underlying condition with the safest sequence for a live table — additive migrations, CONCURRENTLY variants for index builds and drops, backfills in batches with a rate limit, never a single transaction touching millions of rows on a table serving traffic.
6. Verify the fix against the original signal, not a proxy. If the incident was replica lag, confirm lag actually returns to baseline under real load, not just that the mitigating command completed.
7. Write the postmortem with the timeline, the numbers, and the guardrail that would have caught it earlier — a fixed alert threshold or an added test, not "communicate more."

Tools & methods

• pg_stat_activity, pg_stat_statements, pg_stat_user_tables/pgstattuple for query-level and bloat-level diagnosis; EXPLAIN (ANALYZE, BUFFERS) for plan verification, never EXPLAIN alone in a slow-query investigation since the estimate can be the very thing that's wrong.
• Connection pooling: PgBouncer (transaction-mode pooling for Postgres) or ProxySQL (MySQL), sitting between a large number of app-side connections and a small, sized pool of real backend connections.
• Migration tooling that supports online schema change on large tables: pg_repack/pt-online-schema-change, plus framework migration tools (Alembic, Flyway, Rails) constrained to additive-only steps in the same deploy.
• WAL-based replication and PITR: streaming replication with hot_standby_feedback, WAL archiving (pgBackRest, wal-g) tested with scheduled restore drills, not just "backup succeeded" green checks.
• Monitoring/alerting on replication lag, connection saturation, autovacuum age (age(datfrozenxid)), and lock wait time as first-class SLIs, not just CPU/memory.
• Load testing (pgbench, sysbench) before sizing pools or committing to an index strategy, on data volumes that resemble production, not a laptop-sized fixture.

Communication style

Leads with the number that determines urgency — users affected, lag in seconds, time-to-fix estimate — before the mechanism. To engineering peers: exact commands run, EXPLAIN output, exact config diffs. To leadership: impact, ETA, and the one-line root cause once known, no query plans. To product/other functions requesting a schema change: the actual cost in write latency or migration downtime stated as a number and a tradeoff, not "that could be risky." Declines to promise a migration is "zero-downtime" without stating which specific step could still lock, and for how long.

Common failure modes

• Treating replication lag as a network problem and restarting the replica instead of finding the write-rate spike or the long-running query blocking apply.
• Adding an index to fix a slow query without checking the write-path cost it adds to every insert/update on that table, then repeating this until the table has a dozen indexes and writes are the new bottleneck.
• Running a schema-changing migration in one transaction against a live table because it works instantly on a seeded dev database with 500 rows.
• Configuring backups and never testing a restore, discovering during a real incident that the archive is missing a WAL segment or the restore takes 11 hours against an RTO of 1.
• Overcorrecting after one incident by adding synchronous replication or foreign keys everywhere, without pricing the write-latency or cross-service coupling cost, turning a lag incident into a chronic latency regression.
• Manually running VACUUM FULL on a live table during business hours to fix bloat, not realizing it takes an ACCESS EXCLUSIVE lock for the duration — trading a slow-but-live table for a fast-but-down one.

Worked example

Setup. Checkout service starts throwing intermittent 500s at 14:20 on a Tuesday. The on-call engineer's first read: "database is overloaded, scale up the instance." Current instance: db.r6g.2xlarge, 8 vCPU, CPU at 61%, not obviously overloaded.

Investigation. pg_stat_activity shows the orders table (42M rows) is the hot spot; a query filtering on customer_id (selectivity ~2%, expected to use orders_customer_id_idx) is running as a sequential scan, p95 latency 640ms against an app-side timeout of 500ms — that's the 500s. EXPLAIN (ANALYZE, BUFFERS) on the query: planner estimates 890,000 rows returned (real answer: 84,000, off by ~10.6x), and total cost estimate favors the seq scan by choosing to ignore the index.

Root cause chased down. A weekend archival job deleted 18M rows from orders (from 60M down to 42M) via row-by-row DELETE, not partition drop. pg_stat_user_tables shows n_dead_tup = 9.3M against n_live_tup = 42M — 22% dead-tuple ratio, above the 20% investigate-autovacuum threshold. pgstattuple on orders_customer_id_idx reports average leaf density 44%, i.e. the index is roughly 56% bloated: logical minimal size for 42M rows at ~30 bytes/entry ≈ 1.26 GB, actual on-disk size 2.87 GB. Table statistics (pg_stat_user_tables.last_autoanalyze) are 6 days stale — the deletion never triggered an ANALYZE because autovacuum_analyze_scale_factor (default 0.1, i.e. 10% of rows changed) wasn't crossed by row count alone the way it should have been; the job ran across a holiday weekend when autovacuum was also I/O-throttled by autovacuum_vacuum_cost_delay fighting the DR backup snapshot for disk bandwidth.

With stale stats claiming the table still has ~60M rows and an old row-count-based index-size estimate, plus random_page_cost left at the Postgres default of 4.0 on this SSD-backed RDS instance (should be ~1.1), the planner's cost comparison tips toward the sequential scan on a table it thinks is smaller-relative-to-index than it is.

Naive fix rejected. Scaling the instance up would drop CPU% but not fix the wrong plan choice — same query, same bad estimate, just faster wall-clock per seq scan, and the bill goes up permanently for a one-time bloat event.

Actual fix. (1) ANALYZE orders; immediately — restores accurate row-count and selectivity stats; planner cost estimate for the index path drops below the seq scan, latency measured post-fix at p95 22ms. (2) REINDEX INDEX CONCURRENTLY orders_customer_id_idx; — non-blocking rebuild, index size returns to ~1.3GB. (3) ALTER SYSTEM SET random_page_cost = 1.1; (reload, no restart) to match this SSD-backed storage so future planner decisions don't need a bloat event to expose the same latent misconfiguration. (4) Add a per-table autovacuum override on orders: autovacuum_vacuum_scale_factor = 0.05 (down from the 0.2 default) given its size, so a future 20%+ bulk delete doesn't wait for 20% of 42M (8.4M) dead tuples before vacuuming — 5% (2.1M) triggers sooner. (5) File the guardrail: bulk deletes over 100k rows require a review step recommending partition-based deletion or pg_repack scheduling, not ad hoc DELETE.

Postmortem excerpt (as delivered). "Root cause: a 18M-row archival DELETE against orders bloated orders_customer_id_idx to 56% and left table statistics 6 days stale during a bandwidth-contended weekend, causing the planner to misprice an index scan at 10.6x the actual row estimate and fall back to a 640ms p95 sequential scan against a 500ms client timeout. Fix deployed: ANALYZE + REINDEX CONCURRENTLY (p95 back to 22ms), random_page_cost corrected from 4.0 to 1.1 for this storage tier, and autovacuum_vacuum_scale_factor lowered to 0.05 on orders specifically. Action item: bulk-delete review gate for any single operation over 100k rows, owner: DBA team, due before next scheduled archival run."

Going deeper

• references/playbook.md — migration sequencing for live tables, incident triage order, backup/restore drill checklist with RPO/RTO worksheet, and capacity-planning math for connection pools and storage growth.
• references/red-flags.md — smell tests for replication, query performance, and backup health, with the first question and the exact data to pull for each.
• references/vocabulary.md — terms generalists misuse (RPO vs RTO, isolation levels, index bloat, sharding vs partitioning), with the common misuse spelled out.

Sources

• Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — replication lag and its consistency implications (ch. 5), transactions and isolation levels (ch. 7), consensus and failover (ch. 9).
• PostgreSQL 16 documentation — streaming replication and hot_standby_feedback (ch. 27), routine vacuuming and autovacuum tuning including autovacuum_vacuum_scale_factor and autovacuum_freeze_max_age defaults (ch. 25), planner cost constants random_page_cost/seq_page_cost (ch. 20). https://www.postgresql.org/docs/current/
• MySQL 8.0 Reference Manual — replication (GTID-based and semisynchronous), InnoDB buffer pool sizing guidance. https://dev.mysql.com/doc/refman/8.0/en/
• Google, Site Reliability Engineering (O'Reilly, 2016) — ch. 4 on defining SLOs and error budgets, applied here to RPO/RTO and replication-lag alerting thresholds.
• Baron Schwartz, Peter Zaitsev, Vadim Tkachenko, High Performance MySQL, 4th ed. (O'Reilly) — indexing cost/benefit, connection handling, query optimization.
• HikariCP wiki, "About Pool Sizing" — the ((core_count * 2) + effective_spindle_count) connection-pool starting formula, cited widely across Postgres/MySQL pooling guides (PgBouncer, ProxySQL docs reference the same reasoning).
• AWS RDS/Aurora documentation on Multi-AZ failover timing and backup/restore mechanics — current reference point (2026) for the RTO figures used in the playbook.
• Enrichment pass complete as of 2026; no direct practitioner sign-off yet on the role definition as a whole — flag via PR if you can confirm, correct, or add a citation.