Hanuman

Hanuman — Who Carried a Mountain (Migrations & Big Jobs)

Hanuman governs the heavy lifts: move the mountain without dropping it.

Schema migrations

• Every migration is reversible: write and test the down migration (Alembic downgrade, Prisma/Knex down). "Irreversible" migrations need written sign-off and a backup taken first.
• Expand-contract for any breaking change: (1) expand — add the new column/table, code writes both; (2) migrate — backfill old rows; (3) contract — switch reads, drop the old column in a later release. Never rename or drop in one step.
• Never migrate schema and deploy dependent code in the same step. Migration ships first and must be safe with the old code still running.
• No long locks on production: avoid table rewrites; add indexes CONCURRENTLY; add columns as nullable (or with defaults only on Postgres 11+ semantics).
• Migrations are code: reviewed in PRs, run by CI/CD against staging before prod (see brahma).

Backfills and batch jobs

• Idempotent batches: keyed by stable IDs, safe to re-run any batch without double-processing (upsert or skip-if-done).
• Checkpoint progress (last processed ID/offset) in a table or durable store, so a crash resumes — never restarts from zero.
• Dry-run mode first: --dry-run prints what would change (counts, sample rows) without writing. Run it, read it, then run for real.
• Rate-limit against production DBs: small batches (500–5000 rows), sleep between batches, watch replication lag and p95 latency while it runs. Run off-peak.
• Progress logging every batch: processed count, error count, rate, ETA. A silent 6-hour job is a hung job.
• Failures go to a dead-letter list for retry — one bad row must not kill the run.

Verification

• Count before, count after, and reconcile: row counts, sums/checksums on key columns, spot-check N random migrated records against the source.
• Keep the source data until verification passes. Deletion is a separate, later step.

AI-native specifics

• Bulk LLM processing is a batch job: idempotent, checkpointed per record, resumable — API errors and rate limits are guaranteed at scale, plan for them.
• Estimate cost before the run: records × avg tokens × price. Get the number approved before launching (see lakshmi); use provider batch APIs (~50% cheaper) when latency allows.
• Sample-validate first: run 50–100 records, review outputs against the eval criteria (see agni), then launch the full run.
• Store raw model responses alongside parsed results, so a parsing bug means re-parse, not re-spend.
• Pin the model version for the whole run — a mid-run model swap gives inconsistent data.

Before the heavy lift — checklist

• Down migration written and tested; backup exists
• Expand-contract plan for breaking schema changes
• Dry run executed and output reviewed
• Batches idempotent, checkpointed, rate-limited
• Before/after counts reconciled; source retained
• LLM jobs: cost estimated and sample validated first