Matsya

Matsya — Who Saved the World from the Flood (Backups & DR)

Matsya carries what matters through the flood: when everything else drowns, the backups survive.

Backing up

• Automated daily backups plus point-in-time recovery (WAL/binlog) for every prod database. Manual backups are forgotten backups.
• Follow 3-2-1: 3 copies of the data, on 2 different media/systems, 1 offsite (different region or provider).
• Backup before every schema migration, without exception (see hanuman). A migration without a backup is a bet with the whole database.
• Encrypt backups at rest and in transit; backup buckets hold the crown jewels and deserve the same access control as prod (see kubera).
• Use immutable/versioned object storage (S3 versioning + Object Lock) so ransomware or a bad script can't delete the backups along with the data.
• Set retention explicitly: e.g. daily for 30 days, weekly for 6 months, in line with compliance needs. Retention is a decision, not a default.
• Alert on backup job failures. A silently failing backup cron is the classic data-loss story — monitor it like any prod job (see surya).

Restoring

• An untested backup is not a backup. Test restores quarterly: restore into a scratch environment, verify row counts and a sample of critical records, time the process.
• Define RTO (how long to recover) and RPO (how much data loss is acceptable) per system, written down. Prod DB might be RTO 1h / RPO 5min; analytics warehouse can be looser.
• Document the restore runbook step-by-step — exact commands, credentials location, verification steps — so it works at 3 AM under a SEV1 (see dhanvantari), executed by whoever is on call.
• After any restore, verify application behavior against the restored data before declaring recovery, and record what data window (if any) was lost.

Beyond the database

• Back up everything needed to rebuild the system: infra-as-code state (Terraform state with versioning), object storage buckets (cross-region replication), CI/CD and app configuration.
• Code lives in git, but git hosting can fail too — a mirror or periodic repo export is cheap insurance.
• Run a DR game day annually: simulate losing the primary region or database end-to-end and time the recovery against RTO.

AI-native specifics

• Back up the AI stack's real assets: prompt registry, eval datasets, and fine-tuned model weights. Losing the eval set means losing the ability to know if the product still works (see agni).
• Embeddings and vector indexes are rebuildable artifacts — don't burn backup budget on them. Back up the source documents and the pipeline config (chunking, model, parameters) needed to regenerate them, and know how long a rebuild takes.

Backup posture — checklist

• Daily automated backups + PITR on prod DBs, with failure alerts
• 3-2-1 satisfied; backups encrypted and in immutable storage
• Restore tested within the last quarter, timed against RTO
• RTO/RPO written down per system; restore runbook current
• Prompts, evals, fine-tuned weights, and source docs covered
• Backup taken before the last migration; DR game day done this year