Data Loss Incident Response

Data Loss Incident Response

Data Type: {{ data_type }} Description: {{ incident_description }} System: {{ database_system }}

CRITICAL: Stop the Bleeding First

Before investigating, prevent further data loss:

1. STOP any running migrations, scripts, or batch jobs that may be causing the loss
2. BLOCK write access to affected tables/collections/buckets if safe to do so
3. DO NOT attempt recovery before understanding the scope — you may overwrite good data with bad data

Phase 1 — Immediate Containment (0-15 min)

• Identify the source of data loss (accidental deletion, bug, migration, corruption)
• Stop the process causing data loss (kill query, halt migration, disable service)
• Assess if data loss is still ongoing or has stopped
• Declare incident and assign severity
• Prevent further writes to affected data stores if needed

Severity Classification for Data Loss

Phase 2 — Impact Assessment (15-60 min)

Scope the Loss

• How many records/objects/rows are affected?
• What time range of data is affected?
• Which customers/users are impacted?
• Is the lost data reproducible from other sources?
• Are there downstream systems with copies of the data?

Data Classification

Phase 3 — Recovery

Recovery Options (in order of preference)

Option 1: Point-in-Time Recovery (PITR)

• Identify the exact timestamp before the data loss event
• Verify PITR is available and covers the needed time range
• Restore to a SEPARATE instance (never overwrite production)
• Extract only the lost data from the restored instance
• Merge recovered data back into production

Option 2: Restore from Backup

• Identify the most recent backup before the data loss
• Verify backup integrity (checksums, test restore)
• Restore backup to a separate instance
• Identify and extract only the missing data
• Apply any transactions that occurred between backup and loss event
• Merge recovered data into production

Option 3: Recover from Replicas

• Check if read replicas still have the data (replication lag may help)
• Check if any replica was paused or delayed
• Extract data from replica before it catches up to the deletion

Option 4: Reconstruct from Alternative Sources

• Check application-level caches
• Check CDN caches
• Check data warehouse/analytics copies
• Check audit logs for original data values
• Check event streams/message queues for replay
• Contact customers for re-submission (last resort)

Recovery Execution Checklist

• Create recovery plan document before executing
• Get approval from IC and data owner
• Restore to staging/test environment first
• Validate recovered data integrity
• Perform recovery in production
• Verify data counts match expected values
• Run application-level consistency checks

Phase 4 — Data Integrity Verification

• Compare record counts: before vs. after
• Verify referential integrity (foreign keys, cross-references)
• Check for duplicate records introduced during recovery
• Validate data types and constraints
• Run business logic validation queries
• Check downstream systems for consistency
• Verify search indexes are updated
• Confirm cache invalidation completed

Phase 5 — Communication

Internal

• Update incident channel with recovery status
• Notify affected service owners
• Inform data governance team

Customer Communication (if customer data affected)

• Determine which customers lost data
• Quantify what each customer lost
• Notify affected customers with:
  • What data was lost
  • What has been recovered
  • What cannot be recovered
  • What they need to do (re-enter, re-upload, etc.)
  • Preventive measures being taken

Phase 6 — Prevention

Root Cause Remediation

• Implement safeguards against the specific failure mode
• Add confirmation prompts for destructive operations
• Implement soft-delete instead of hard-delete where possible
• Add transaction logging for all write operations

Backup and Recovery Improvements

• Verify backup schedule meets RPO requirements
• Test backup restoration regularly (quarterly minimum)
• Enable PITR if not already active
• Implement cross-region backup replication
• Add backup integrity monitoring

Counter-Rationalizations