Amey-Thakur Disaster recovery testing
An untested backup is a hope. Most recovery failures are discovered during the first real attempt: incomplete backups, missing credentials, procedures referencing systems that no longer exist.
Method
- Restore from backup on a schedule. Actually restoring is the only verification, since a successful backup job proves nothing about restorability (see backup-restore).
- Measure the real recovery time. How long a full restore takes, compared to the objective you have promised, which is usually a surprise the first time.
- Practise the whole path. Access, credentials, DNS, dependencies, and data, because recovery fails at the step nobody documented.
- Rotate who runs the drill. A procedure only one person can execute has not been tested, and that person will be unavailable.
- Test failover in both directions. Failing back is often harder than failing over and is rarely rehearsed.
- Verify data integrity after restore, not just completion. A restore that produces corrupt or partial data has failed silently.
- Update the runbook from what you learned. Every drill finds inaccuracies, and correcting them is the output (see runbook-writing).
Boundaries
Drills test the scenarios you rehearse and not the one that happens. Full production drills carry real risk and need careful scoping. Recovery objectives that the tested time cannot meet are commitments you should correct rather than restate.