qa-aman Overview
Based on "The Checklist Manifesto" by Atul Gawande and "Accelerate" by Forsgren, Humble & Kim.
Gawande's insight: checklists don't exist because people are stupid - they exist because complex, high-pressure situations cause smart people to skip steps they know matter. A pre-deployment checklist is not bureaucracy; it's the discipline that separates teams with 15% change failure rates from teams with 60%.
Accelerate's finding: high-performing teams deploy more frequently AND have lower failure rates. The checklist is what enables speed without sacrificing safety.
Workflow
Pre-Deployment Checklist
Run before every production deployment:
Code and testing:
- CI pipeline is green (all tests pass, no warnings suppressed)
- Code reviewed and approved by at least one other engineer
- No
TODO: fix before deploycomments in the diff - No hardcoded credentials, secrets, or API keys introduced
- Database migrations are backward-compatible (old code can run against new schema)
Deployment mechanics:
- Deployment plan documented: what is being deployed, to which environment, in what order
- Rollback plan documented: exact steps to revert if something goes wrong
- Feature flags configured (if applicable): new code is behind a flag for gradual rollout
- Deployment window confirmed: not during peak traffic hours or known high-risk periods
Dependencies and coordination:
- Dependent teams notified if this change affects shared APIs or contracts
- On-call engineer aware that a deployment is happening
- Monitoring dashboards open and baseline metrics noted before deploy begins
Data and migrations:
- Database migrations tested on a staging environment with production-scale data
- Data migration has a rollback path (or data is backed up)
- If schema change: old and new code can run simultaneously during deploy window
Deployment Execution
During the deploy:
- Note start time and deployer in the incident channel / deploy log
- Monitor error rate and latency during and after deploy (minimum 10 minutes)
- Verify key user journeys manually on production after deploy
- Note end time and result
Deploy log entry:
Date/time: [start] to [end]
Deployer: [name]
Change: [what was deployed, link to PR/ticket]
Result: [Success / Rolled back]
Notes: [anything unusual observed]
Post-Deployment Checklist
Run within 30 minutes of deployment:
- Error rate is at or below pre-deployment baseline
- p99 latency is at or below pre-deployment baseline
- Key user journeys verified manually (login, core feature, critical path)
- No new alerts firing
- Logs are clean (no unexpected errors in the first 10 minutes)
- Feature flag enabled for target percentage (if gradual rollout)
- Deployment logged with result, time, and deployer
Rollback Triggers
Roll back immediately if:
- Error rate > 2x pre-deployment baseline for more than 2 minutes
- P0/P1 alert fires that was not present before deployment
- Core user journey is broken
- Data corruption is detected
Do not wait to diagnose before rolling back. Restore first, investigate after.
Anti-Patterns
1. Skipping the checklist under time pressure Bad: "We need to deploy this hotfix now, no time for checks." Good: A 5-minute checklist prevents a 2-hour incident. The checklist is fastest under pressure, not slowest.
2. Deploying during peak traffic Bad: Deploying at 2pm on a Friday. Good: Deploy during low-traffic windows. Know your traffic patterns before setting deployment windows.
3. No rollback plan Bad: "If something goes wrong, we'll figure it out." Good: Rollback steps documented before every deploy. Tested quarterly.
4. Checking boxes without checking Bad: Marking checklist items as done without actually verifying them. Good: Gawande's insight: the checklist is a discipline tool, not a paperwork exercise. Each box represents a real check.
Quality Checklist
- Pre-deployment checklist completed and all items pass
- Rollback plan documented before deployment begins
- On-call engineer aware of deployment
- Monitoring dashboards open with baseline metrics noted
- Post-deployment monitoring ran for at least 10 minutes
- Post-deployment checklist completed
- Deploy log entry written with result
- Rollback triggers defined in advance