Varuna

Varuna — Lord of the Oceans (Databases & Data Stores)

Varuna governs the waters where data lives: one well-kept ocean beats a dozen puddles.

Choosing a store

• Postgres by default. It handles relational, JSONB documents, full-text search, and vectors (pgvector) — earn your way out of it, don't start out of it.
• Choose by access pattern, not fashion: relational (transactions, joins) → Postgres; hot KV/cache/queues → Redis; true document scale-out → only with a proven need; vector search → pgvector first.
• Every additional store is an operational tax: backups, monitoring, migrations, one more thing to page on.

Schema & migrations

• All schema changes via a migration tool — Alembic (Python/SQLAlchemy) or Prisma Migrate (TS). Never hand-run DDL in prod.
• Migrations are reversible where possible, reviewed in PR, applied by CI/deploy — same flow as code.
• Every foreign key gets an index (Postgres does not auto-index FKs). Every column in a frequent WHERE/ORDER BY earns index consideration.
• Use NOT NULL + defaults + constraints in the schema; don't rely on app code for integrity. Timestamps: timestamptz, always UTC.

Queries

• No SELECT * in application code — name your columns; schema changes shouldn't silently change payloads.
• EXPLAIN ANALYZE any query > 100ms before "fixing" it — measure, then index. Watch for seq scans on large tables.
• Prevent N+1: use joins/selectinload (SQLAlchemy) or include (Prisma); log query counts per request in dev.
• Paginate with keyset (cursor) pagination for large tables, not OFFSET.

Connections & transactions

• Connection pooling always: pgbouncer (transaction mode) in front of Postgres for serverless/many-worker setups; size pools deliberately.
• Keep transactions short — no network calls (and never an LLM call) inside an open transaction.
• Set statement timeouts so a runaway query can't hold the ocean hostage.

Deletion & lifecycle

• Soft-delete (deleted_at) only when undo/audit truly requires it — and then filter it in one place (default scope/view), index it, and plan a hard-purge job. Otherwise delete for real.
• Backups, PITR, and restore drills belong to matsya — but no new store ships without them.

AI-native specifics

• pgvector before a dedicated vector DB. Move only when scale (tens of millions of vectors, heavy filtering) proves the need.
• Store with every embedding: model name, model version, dimension, and source content hash — mixed-model embeddings are silent garbage.
• Have a re-embed strategy before changing embedding models: dual-write or batch re-embed, verify retrieval evals, then cut over.
• Document chunking decisions (size, overlap, splitter) next to the ingestion code; changing chunking means re-embedding.

Before shipping data changes — checklist

• Schema change is a reviewed migration, reversible where possible
• FKs and hot query paths indexed; slow queries EXPLAIN ANALYZEd
• No SELECT *; pooling via pgbouncer; transactions short
• Embeddings carry model + version metadata; re-embed plan exists
• Backups/restore covered (see matsya) before the store takes prod traffic