Pipeline Safety
Overview
A safe pipeline is one that makes it harder to deploy bad code than good code. It enforces automated quality gates at every stage, blocks deployments when alarms are firing, provides one-click rollback, and creates an audit trail of every production change. The pipeline is not a convenience tool—it is a safety system that protects customers.
When to Use
- Setting up a new service's CI/CD pipeline
- Adding safety gates to an existing pipeline
- Reviewing pipeline configuration for operational readiness
- After any incident caused by a deployment that should have been caught
- When a team is deploying less frequently due to fear of breaking things (symptom of insufficient safety)
Amazon Context
Amazon's deployment philosophy: the pipeline should be the safest, fastest path to production. If developers feel tempted to bypass the pipeline, the pipeline is broken—either too slow or too fragile. A good pipeline deploys hundreds of times per day across a large organization while catching the handful of changes that would have caused customer impact. The investment in pipeline safety is what enables high deployment velocity.
The Process
Pre-Commit Gates
- Static analysis: Linting, type checking, security scanning (SAST)
- Unit tests: Must pass 100%. No "known failures" or skipped tests without expiration dates
- Dependency check: No known vulnerabilities in dependencies above configured severity
- Commit message format: Structured messages that link to tickets/tasks
Pre-Merge Gates
- Integration tests: Service-level integration test suite
- Code coverage: Cannot decrease. New code must meet minimum threshold (e.g., 80%)
- Code review approval: At least one approval from a qualified reviewer
- Design review link: For changes above a size threshold, link to approved design
Pre-Deploy Gates
- Alarm check: No active alarms on the target service (P1, P2 block; P3 warn)
- Deployment window: Respect maintenance windows and blackout periods
- Active incident check: No ongoing incidents for this service or critical dependencies
- Rollback verification: Confirm rollback target exists and is healthy
- Capacity check: Sufficient capacity to handle deployment (rolling deploy won't reduce capacity below safe minimum)
During-Deploy Gates
- Health check: New instances must pass health checks before receiving traffic
- Metric comparison: Real-time comparison against baseline (previous deployment metrics)
- Synthetic canary: Automated customer-journey tests running continuously
- Dependency health: Verify downstream services are healthy
- Automatic pause: Stop deployment if any metric breaches threshold
Post-Deploy Gates
- Bake time enforcement: Pipeline doesn't mark "complete" until bake time passes with green metrics
- Smoke tests: Automated verification of critical paths
- Alarm check (again): Verify no new alarms fired during bake period
- Audit log: Record who deployed what, when, and the metrics observed
One-Click Rollback
Every deployment must support instant rollback:
- Previous known-good artifact is always retained
- Rollback is a single action (button, command, or automatic trigger)
- Rollback does not require a new build or new tests
- Rollback is tested regularly (at least monthly via actual rollback or chaos exercise)
- Rollback time target: <5 minutes from decision to full rollback
Deployment Blockers
The pipeline MUST block deployment when:
- Active P1 or P2 alarm on the service
- Active incident bridge for the service
- Outside deployment window (configurable per-team)
- Required approvals not obtained
- Test suite has failures
- Security scan has critical findings
- Previous deployment is still baking
Override requires: documented justification + senior engineer approval + post-deploy review within 24 hours.
Mechanisms Over Good Intentions
| Intention |
Mechanism |
| "I'll check alarms before deploying" |
Pipeline automatically blocks on active alarms |
| "I'll rollback quickly if something goes wrong" |
Automatic rollback on alarm, not human decision |
| "I'll make sure tests pass" |
Pipeline blocks merge on any test failure |
| "I won't deploy during incidents" |
Pipeline queries incident system and blocks |
| "I'll deploy during business hours" |
Deployment windows enforced by pipeline |
Common Rationalizations
| What They Say |
Why It's Wrong |
What To Do Instead |
| "The pipeline is too slow" |
A pipeline that catches bugs before production saves more time than it costs |
Optimize the pipeline (parallel tests, caching) rather than removing gates |
| "This fix is urgent, I need to skip gates" |
Urgent fixes are exactly when you make mistakes under pressure |
Use the emergency pipeline—fewer gates but never zero |
| "Tests are flaky, we have to ignore them" |
Flaky tests mask real failures. Fix them. |
Mark flaky tests with expiration dates; fix or delete within 2 weeks |
| "The alarm is a false positive" |
You don't know that until you investigate |
Fix the alarm signal, don't deploy over it |
Red Flags
- Developers routinely skip or override pipeline gates
- "Emergency" deployments happen more than once per quarter
- Rollback requires building and deploying a new artifact
- No alarm check before deployment
- Test results are advisory rather than blocking
- No deployment windows or blackout periods
- Pipeline takes >60 minutes for a standard deployment
- No audit trail of who deployed what and when
- Rollback hasn't been tested in the last 30 days
Verification
Tenets
- The pipeline protects customers, not developers' convenience. Every gate exists because a real incident happened without it.
- Fast and safe are not opposites. A well-designed pipeline is both. If yours isn't, fix the pipeline, don't remove safety.
- Blocking is better than warning. Warnings are ignored under pressure. Gates must block.
- Every override is a data point. Track overrides. If they happen often, the pipeline needs improvement.
- Rollback is not failure; inability to rollback is failure. Design every deployment to be reversible.
1---2name: pipeline-safety3description: CI/CD pipeline safety with automated gates, deployment blockers, alarm checks, and one-click rollback.4---56# Pipeline Safety78## Overview910A safe pipeline is one that makes it harder to deploy bad code than good code. It enforces automated quality gates at every stage, blocks deployments when alarms are firing, provides one-click rollback, and creates an audit trail of every production change. The pipeline is not a convenience tool—it is a safety system that protects customers.1112## When to Use1314- Setting up a new service's CI/CD pipeline15- Adding safety gates to an existing pipeline16- Reviewing pipeline configuration for operational readiness17- After any incident caused by a deployment that should have been caught18- When a team is deploying less frequently due to fear of breaking things (symptom of insufficient safety)1920## Amazon Context2122Amazon's deployment philosophy: the pipeline should be the safest, fastest path to production. If developers feel tempted to bypass the pipeline, the pipeline is broken—either too slow or too fragile. A good pipeline deploys hundreds of times per day across a large organization while catching the handful of changes that would have caused customer impact. The investment in pipeline safety is what enables high deployment velocity.2324## The Process2526### Pre-Commit Gates27281. **Static analysis**: Linting, type checking, security scanning (SAST)292. **Unit tests**: Must pass 100%. No "known failures" or skipped tests without expiration dates303. **Dependency check**: No known vulnerabilities in dependencies above configured severity314. **Commit message format**: Structured messages that link to tickets/tasks3233### Pre-Merge Gates34351. **Integration tests**: Service-level integration test suite362. **Code coverage**: Cannot decrease. New code must meet minimum threshold (e.g., 80%)373. **Code review approval**: At least one approval from a qualified reviewer384. **Design review link**: For changes above a size threshold, link to approved design3940### Pre-Deploy Gates41421. **Alarm check**: No active alarms on the target service (P1, P2 block; P3 warn)432. **Deployment window**: Respect maintenance windows and blackout periods443. **Active incident check**: No ongoing incidents for this service or critical dependencies454. **Rollback verification**: Confirm rollback target exists and is healthy465. **Capacity check**: Sufficient capacity to handle deployment (rolling deploy won't reduce capacity below safe minimum)4748### During-Deploy Gates49501. **Health check**: New instances must pass health checks before receiving traffic512. **Metric comparison**: Real-time comparison against baseline (previous deployment metrics)523. **Synthetic canary**: Automated customer-journey tests running continuously534. **Dependency health**: Verify downstream services are healthy545. **Automatic pause**: Stop deployment if any metric breaches threshold5556### Post-Deploy Gates57581. **Bake time enforcement**: Pipeline doesn't mark "complete" until bake time passes with green metrics592. **Smoke tests**: Automated verification of critical paths603. **Alarm check (again)**: Verify no new alarms fired during bake period614. **Audit log**: Record who deployed what, when, and the metrics observed6263### One-Click Rollback6465Every deployment must support instant rollback:66- Previous known-good artifact is always retained67- Rollback is a single action (button, command, or automatic trigger)68- Rollback does not require a new build or new tests69- Rollback is tested regularly (at least monthly via actual rollback or chaos exercise)70- Rollback time target: <5 minutes from decision to full rollback7172### Deployment Blockers7374The pipeline MUST block deployment when:75- Active P1 or P2 alarm on the service76- Active incident bridge for the service77- Outside deployment window (configurable per-team)78- Required approvals not obtained79- Test suite has failures80- Security scan has critical findings81- Previous deployment is still baking8283Override requires: documented justification + senior engineer approval + post-deploy review within 24 hours.8485## Mechanisms Over Good Intentions8687| Intention | Mechanism |88|-----------|-----------|89| "I'll check alarms before deploying" | Pipeline automatically blocks on active alarms |90| "I'll rollback quickly if something goes wrong" | Automatic rollback on alarm, not human decision |91| "I'll make sure tests pass" | Pipeline blocks merge on any test failure |92| "I won't deploy during incidents" | Pipeline queries incident system and blocks |93| "I'll deploy during business hours" | Deployment windows enforced by pipeline |9495## Common Rationalizations9697| What They Say | Why It's Wrong | What To Do Instead |98|---------------|---------------|-------------------|99| "The pipeline is too slow" | A pipeline that catches bugs before production saves more time than it costs | Optimize the pipeline (parallel tests, caching) rather than removing gates |100| "This fix is urgent, I need to skip gates" | Urgent fixes are exactly when you make mistakes under pressure | Use the emergency pipeline—fewer gates but never zero |101| "Tests are flaky, we have to ignore them" | Flaky tests mask real failures. Fix them. | Mark flaky tests with expiration dates; fix or delete within 2 weeks |102| "The alarm is a false positive" | You don't know that until you investigate | Fix the alarm signal, don't deploy over it |103104## Red Flags105106- Developers routinely skip or override pipeline gates107- "Emergency" deployments happen more than once per quarter108- Rollback requires building and deploying a new artifact109- No alarm check before deployment110- Test results are advisory rather than blocking111- No deployment windows or blackout periods112- Pipeline takes >60 minutes for a standard deployment113- No audit trail of who deployed what and when114- Rollback hasn't been tested in the last 30 days115116## Verification117118- [ ] All pre-commit gates are automated and blocking119- [ ] All pre-merge gates are automated and blocking120- [ ] Alarm check runs before every deployment121- [ ] Deployment blockers are enforced (not advisory)122- [ ] One-click rollback exists and has been tested in the last 30 days123- [ ] Pipeline creates an audit log for every deployment124- [ ] Emergency pipeline exists with reduced (not zero) gates125- [ ] Pipeline completes in <30 minutes for standard changes126- [ ] Override requires documented justification and senior approval127- [ ] Deployment windows are configured and enforced128129## Tenets1301311. **The pipeline protects customers, not developers' convenience.** Every gate exists because a real incident happened without it.1322. **Fast and safe are not opposites.** A well-designed pipeline is both. If yours isn't, fix the pipeline, don't remove safety.1333. **Blocking is better than warning.** Warnings are ignored under pressure. Gates must block.1344. **Every override is a data point.** Track overrides. If they happen often, the pipeline needs improvement.1355. **Rollback is not failure; inability to rollback is failure.** Design every deployment to be reversible.
robisson