Production Readiness Review
Comprehensive evaluation of whether a system is ready to serve production traffic safely, reliably, and observably.
Cross-references:
observability/knowledge-observability-pillars for logs, metrics, tracesdeployment/knowledge-release-strategies for rollout and rollback patterns
When to Use
- Before first production deployment of a new service
- Before major releases with significant changes
- After extended outages (validate fixes address root cause)
- Periodic review of production services (quarterly)
Severity Levels
| Level |
Meaning |
| CRITICAL |
Will cause outage, data loss, or security incident — block deployment |
| WARNING |
Increases incident likelihood or response time — fix before next release |
| SUGGESTION |
Operational improvement — implement as capacity allows |
Review Checklist
1. Observability
Can you detect, diagnose, and understand problems in production?
Logging
| Check |
Severity if violated |
| Structured logging (JSON) with consistent field names |
WARNING |
| Request correlation IDs propagated across services |
WARNING |
| Log levels used correctly (ERROR for failures, WARN for degradation, INFO for business events, DEBUG off in prod) |
WARNING |
| No sensitive data in logs (passwords, tokens, full credit card numbers, PII) |
CRITICAL |
| Logs aggregated to central system (ELK, CloudWatch, Datadog) |
CRITICAL |
| Log retention policy configured |
WARNING |
Metrics
| Check |
Severity if violated |
| RED metrics exposed: Rate, Errors, Duration per endpoint |
CRITICAL |
| System metrics: CPU, memory, disk, network |
WARNING |
| Business metrics: key domain events counted |
WARNING |
| SLIs defined and measured (latency p50/p95/p99, availability, error rate) |
WARNING |
| SLOs documented with error budgets |
SUGGESTION |
| Metrics dashboards exist and are accessible to the team |
WARNING |
Tracing
| Check |
Severity if violated |
| Distributed tracing instrumented across service boundaries |
WARNING |
| Trace context propagated in all inter-service calls |
WARNING |
| Sampling rate configured appropriately (not 0%, not 100% in high-traffic) |
SUGGESTION |
Alerting
| Check |
Severity if violated |
| Alerts defined for SLO violations |
CRITICAL |
| Alerts on error rate spikes (not just absolute thresholds) |
WARNING |
| Alert routing configured to on-call channel (PagerDuty, Opsgenie) |
CRITICAL |
| No alert fatigue (alerts are actionable, not noisy) |
WARNING |
| Alerts tested with synthetic failures |
SUGGESTION |
2. Reliability
Can the system handle failures gracefully?
Health and Lifecycle
| Check |
Severity if violated |
Health check endpoint exists (/healthz or /health) |
CRITICAL |
| Readiness probe distinct from liveness probe |
WARNING |
| Graceful shutdown (drain connections, finish in-flight requests) |
CRITICAL |
| Startup probe for slow-starting services |
SUGGESTION |
| Health checks verify actual dependencies, not just process alive |
WARNING |
Failure Handling
| Check |
Severity if violated |
| Circuit breakers on external dependencies |
WARNING |
| Retries with exponential backoff and jitter |
WARNING |
| Timeouts on all outbound calls (HTTP, DB, queue) |
CRITICAL |
| Bulkheads to isolate failure domains (separate thread/connection pools per dependency) |
SUGGESTION |
| Fallback behavior defined for degraded dependencies |
WARNING |
| Idempotency on write operations exposed to retries |
WARNING |
Data Safety
| Check |
Severity if violated |
| Database backups automated and tested (restore drill completed) |
CRITICAL |
| No single point of failure for critical data |
CRITICAL |
| Message queue durability configured (persistent messages, DLQ) |
WARNING |
| Data migration rollback tested |
WARNING |
3. Security
Is the system hardened for production?
| Check |
Severity if violated |
| Secrets in vault/env vars, not in code or config files |
CRITICAL |
| TLS on all external and inter-service communication |
CRITICAL |
| Authentication and authorization enforced on all endpoints |
CRITICAL |
| Dependency vulnerability scan passing (no critical/high CVEs) |
CRITICAL |
| Container runs as non-root user |
WARNING |
| Network policies restrict unnecessary traffic |
WARNING |
| Rate limiting on public endpoints |
WARNING |
| Security headers configured (CSP, HSTS, X-Frame-Options) |
WARNING |
| CORS restrictively configured |
WARNING |
4. Operations
Can the team operate this service day-to-day?
Deployment
| Check |
Severity if violated |
| Deployment is automated (CI/CD, not manual steps) |
CRITICAL |
| Rollback procedure documented and tested (<5 min to roll back) |
CRITICAL |
| Canary or blue-green deployment strategy in place |
WARNING |
| Feature flags for risky changes |
SUGGESTION |
| Deployment doesn't require downtime |
WARNING |
| Smoke tests run post-deployment |
WARNING |
Documentation
| Check |
Severity if violated |
| Runbook exists covering common failure scenarios |
CRITICAL |
| Architecture diagram current |
WARNING |
| API documentation complete and up to date |
WARNING |
| On-call handoff document with escalation paths |
WARNING |
| Known issues / tech debt documented |
SUGGESTION |
Incident Response
| Check |
Severity if violated |
| On-call rotation established |
CRITICAL |
| Team can access production logs, metrics, dashboards |
CRITICAL |
| Incident communication channel defined |
WARNING |
| Post-incident review process defined |
SUGGESTION |
5. Scalability
Can the system handle expected and unexpected load?
| Check |
Severity if violated |
| Load testing performed at 2x expected peak traffic |
WARNING |
| Auto-scaling configured with appropriate min/max |
WARNING |
| Resource requests and limits set (CPU, memory) |
CRITICAL |
| Database can handle projected data volume for 12+ months |
WARNING |
| Connection pools sized for scaled replica count |
WARNING |
| No in-process state that prevents horizontal scaling |
CRITICAL |
| Rate limiting protects against traffic spikes |
WARNING |
| Caching layer can handle cache miss storms |
SUGGESTION |
| Queue consumers scale independently of producers |
SUGGESTION |
6. Compliance and Data
| Check |
Severity if violated |
| PII handling documented and compliant (GDPR, CCPA) |
CRITICAL |
| Data retention policies configured |
WARNING |
| Audit logging for sensitive operations |
WARNING |
| Data classification applied (public, internal, confidential) |
SUGGESTION |
Output Format
## Production Readiness Review: [Service/System Name]
**Version/Release**: [version or PR]
**Review Date**: [date]
**Reviewer**: [agent or person]
**Overall**: [READY | READY WITH CAVEATS | NOT READY]
### Findings
#### Observability
##### [CRITICAL] No alerting on error rate
**Issue**: Error rate has no alert — outages go undetected
**Impact**: MTTR increases, SLO breaches go unnoticed
**Fix**: Add alert when error_rate > 1% for 5 minutes, route to PagerDuty
#### Reliability
##### [CRITICAL] No graceful shutdown
**Issue**: Service kills in-flight requests on deploy
**Impact**: Users see intermittent 502 errors during deploys
**Fix**: Handle SIGTERM, drain connections, finish requests within 30s timeout
...
### Summary
| Category | Critical | Warning | Suggestion |
|----------|----------|---------|------------|
| Observability | 1 | 3 | 1 |
| Reliability | 1 | 2 | 1 |
| Security | 0 | 2 | 0 |
| Operations | 1 | 1 | 1 |
| Scalability | 0 | 2 | 1 |
| Compliance | 0 | 1 | 0 |
| **Total** | **3** | **11** | **4** |
### Production Readiness Score
| Category | Score |
|----------|-------|
| Observability | 🟡 Partial |
| Reliability | 🔴 Gaps |
| Security | 🟢 Good |
| Operations | 🟡 Partial |
| Scalability | 🟢 Good |
| Compliance | 🟡 Partial |
### Launch Blockers (must fix)
1. <Critical items that block deployment>
2. ...
### Post-Launch (fix within 30 days)
1. <Warning items>
2. ...
### Improvement Backlog
1. <Suggestions>
2. ...
Quick Pre-Launch Gate
For a fast go/no-go decision, these are the absolute minimum requirements:
| # |
Requirement |
Status |
| 1 |
Health check endpoint works |
☐ |
| 2 |
Graceful shutdown implemented |
☐ |
| 3 |
Logs aggregated and searchable |
☐ |
| 4 |
Alerts on error rate and latency |
☐ |
| 5 |
Secrets not in code |
☐ |
| 6 |
TLS everywhere |
☐ |
| 7 |
Auth on all endpoints |
☐ |
| 8 |
Rollback tested |
☐ |
| 9 |
Runbook exists |
☐ |
| 10 |
On-call rotation set |
☐ |
All 10 must be checked for a GO decision.
1---2name: review-production-readiness3description: Comprehensive assessment of whether a system is ready for production traffic. Covers observability, reliability, security, operations, and scalability. The most thorough reviewer — use before launch or major releases.4---56# Production Readiness Review78Comprehensive evaluation of whether a system is ready to serve production traffic safely, reliably, and observably.910**Cross-references**:11- `observability/knowledge-observability-pillars` for logs, metrics, traces12- `deployment/knowledge-release-strategies` for rollout and rollback patterns1314## When to Use1516- Before first production deployment of a new service17- Before major releases with significant changes18- After extended outages (validate fixes address root cause)19- Periodic review of production services (quarterly)2021## Severity Levels2223| Level | Meaning |24|-------|---------|25| **CRITICAL** | Will cause outage, data loss, or security incident — block deployment |26| **WARNING** | Increases incident likelihood or response time — fix before next release |27| **SUGGESTION** | Operational improvement — implement as capacity allows |2829## Review Checklist3031### 1. Observability3233_Can you detect, diagnose, and understand problems in production?_3435#### Logging3637| Check | Severity if violated |38|-------|---------------------|39| Structured logging (JSON) with consistent field names | WARNING |40| Request correlation IDs propagated across services | WARNING |41| Log levels used correctly (ERROR for failures, WARN for degradation, INFO for business events, DEBUG off in prod) | WARNING |42| No sensitive data in logs (passwords, tokens, full credit card numbers, PII) | CRITICAL |43| Logs aggregated to central system (ELK, CloudWatch, Datadog) | CRITICAL |44| Log retention policy configured | WARNING |4546#### Metrics4748| Check | Severity if violated |49|-------|---------------------|50| RED metrics exposed: Rate, Errors, Duration per endpoint | CRITICAL |51| System metrics: CPU, memory, disk, network | WARNING |52| Business metrics: key domain events counted | WARNING |53| SLIs defined and measured (latency p50/p95/p99, availability, error rate) | WARNING |54| SLOs documented with error budgets | SUGGESTION |55| Metrics dashboards exist and are accessible to the team | WARNING |5657#### Tracing5859| Check | Severity if violated |60|-------|---------------------|61| Distributed tracing instrumented across service boundaries | WARNING |62| Trace context propagated in all inter-service calls | WARNING |63| Sampling rate configured appropriately (not 0%, not 100% in high-traffic) | SUGGESTION |6465#### Alerting6667| Check | Severity if violated |68|-------|---------------------|69| Alerts defined for SLO violations | CRITICAL |70| Alerts on error rate spikes (not just absolute thresholds) | WARNING |71| Alert routing configured to on-call channel (PagerDuty, Opsgenie) | CRITICAL |72| No alert fatigue (alerts are actionable, not noisy) | WARNING |73| Alerts tested with synthetic failures | SUGGESTION |7475### 2. Reliability7677_Can the system handle failures gracefully?_7879#### Health and Lifecycle8081| Check | Severity if violated |82|-------|---------------------|83| Health check endpoint exists (`/healthz` or `/health`) | CRITICAL |84| Readiness probe distinct from liveness probe | WARNING |85| Graceful shutdown (drain connections, finish in-flight requests) | CRITICAL |86| Startup probe for slow-starting services | SUGGESTION |87| Health checks verify actual dependencies, not just process alive | WARNING |8889#### Failure Handling9091| Check | Severity if violated |92|-------|---------------------|93| Circuit breakers on external dependencies | WARNING |94| Retries with exponential backoff and jitter | WARNING |95| Timeouts on all outbound calls (HTTP, DB, queue) | CRITICAL |96| Bulkheads to isolate failure domains (separate thread/connection pools per dependency) | SUGGESTION |97| Fallback behavior defined for degraded dependencies | WARNING |98| Idempotency on write operations exposed to retries | WARNING |99100#### Data Safety101102| Check | Severity if violated |103|-------|---------------------|104| Database backups automated and tested (restore drill completed) | CRITICAL |105| No single point of failure for critical data | CRITICAL |106| Message queue durability configured (persistent messages, DLQ) | WARNING |107| Data migration rollback tested | WARNING |108109### 3. Security110111_Is the system hardened for production?_112113| Check | Severity if violated |114|-------|---------------------|115| Secrets in vault/env vars, not in code or config files | CRITICAL |116| TLS on all external and inter-service communication | CRITICAL |117| Authentication and authorization enforced on all endpoints | CRITICAL |118| Dependency vulnerability scan passing (no critical/high CVEs) | CRITICAL |119| Container runs as non-root user | WARNING |120| Network policies restrict unnecessary traffic | WARNING |121| Rate limiting on public endpoints | WARNING |122| Security headers configured (CSP, HSTS, X-Frame-Options) | WARNING |123| CORS restrictively configured | WARNING |124125### 4. Operations126127_Can the team operate this service day-to-day?_128129#### Deployment130131| Check | Severity if violated |132|-------|---------------------|133| Deployment is automated (CI/CD, not manual steps) | CRITICAL |134| Rollback procedure documented and tested (<5 min to roll back) | CRITICAL |135| Canary or blue-green deployment strategy in place | WARNING |136| Feature flags for risky changes | SUGGESTION |137| Deployment doesn't require downtime | WARNING |138| Smoke tests run post-deployment | WARNING |139140#### Documentation141142| Check | Severity if violated |143|-------|---------------------|144| Runbook exists covering common failure scenarios | CRITICAL |145| Architecture diagram current | WARNING |146| API documentation complete and up to date | WARNING |147| On-call handoff document with escalation paths | WARNING |148| Known issues / tech debt documented | SUGGESTION |149150#### Incident Response151152| Check | Severity if violated |153|-------|---------------------|154| On-call rotation established | CRITICAL |155| Team can access production logs, metrics, dashboards | CRITICAL |156| Incident communication channel defined | WARNING |157| Post-incident review process defined | SUGGESTION |158159### 5. Scalability160161_Can the system handle expected and unexpected load?_162163| Check | Severity if violated |164|-------|---------------------|165| Load testing performed at 2x expected peak traffic | WARNING |166| Auto-scaling configured with appropriate min/max | WARNING |167| Resource requests and limits set (CPU, memory) | CRITICAL |168| Database can handle projected data volume for 12+ months | WARNING |169| Connection pools sized for scaled replica count | WARNING |170| No in-process state that prevents horizontal scaling | CRITICAL |171| Rate limiting protects against traffic spikes | WARNING |172| Caching layer can handle cache miss storms | SUGGESTION |173| Queue consumers scale independently of producers | SUGGESTION |174175### 6. Compliance and Data176177| Check | Severity if violated |178|-------|---------------------|179| PII handling documented and compliant (GDPR, CCPA) | CRITICAL |180| Data retention policies configured | WARNING |181| Audit logging for sensitive operations | WARNING |182| Data classification applied (public, internal, confidential) | SUGGESTION |183184## Output Format185186```markdown187## Production Readiness Review: [Service/System Name]188189**Version/Release**: [version or PR]190**Review Date**: [date]191**Reviewer**: [agent or person]192**Overall**: [READY | READY WITH CAVEATS | NOT READY]193194### Findings195196#### Observability197##### [CRITICAL] No alerting on error rate198**Issue**: Error rate has no alert — outages go undetected199**Impact**: MTTR increases, SLO breaches go unnoticed200**Fix**: Add alert when error_rate > 1% for 5 minutes, route to PagerDuty201202#### Reliability203##### [CRITICAL] No graceful shutdown204**Issue**: Service kills in-flight requests on deploy205**Impact**: Users see intermittent 502 errors during deploys206**Fix**: Handle SIGTERM, drain connections, finish requests within 30s timeout207208...209210### Summary211| Category | Critical | Warning | Suggestion |212|----------|----------|---------|------------|213| Observability | 1 | 3 | 1 |214| Reliability | 1 | 2 | 1 |215| Security | 0 | 2 | 0 |216| Operations | 1 | 1 | 1 |217| Scalability | 0 | 2 | 1 |218| Compliance | 0 | 1 | 0 |219| **Total** | **3** | **11** | **4** |220221### Production Readiness Score222223| Category | Score |224|----------|-------|225| Observability | 🟡 Partial |226| Reliability | 🔴 Gaps |227| Security | 🟢 Good |228| Operations | 🟡 Partial |229| Scalability | 🟢 Good |230| Compliance | 🟡 Partial |231232### Launch Blockers (must fix)2331. <Critical items that block deployment>2342. ...235236### Post-Launch (fix within 30 days)2371. <Warning items>2382. ...239240### Improvement Backlog2411. <Suggestions>2422. ...243```244245## Quick Pre-Launch Gate246247For a fast go/no-go decision, these are the absolute minimum requirements:248249| # | Requirement | Status |250|---|-------------|--------|251| 1 | Health check endpoint works | ☐ |252| 2 | Graceful shutdown implemented | ☐ |253| 3 | Logs aggregated and searchable | ☐ |254| 4 | Alerts on error rate and latency | ☐ |255| 5 | Secrets not in code | ☐ |256| 6 | TLS everywhere | ☐ |257| 7 | Auth on all endpoints | ☐ |258| 8 | Rollback tested | ☐ |259| 9 | Runbook exists | ☐ |260| 10 | On-call rotation set | ☐ |261262**All 10 must be checked for a GO decision.**
DeanKelly751