Deployment Strategies

Deployment Strategies

When to Use

Use this skill when:

• User asks which deployment strategy (blue/green, canary, rolling, recreate, A/B, shadow) fits their application and infrastructure
• User needs to implement or improve a zero-downtime deployment pipeline for a production service
• User wants to reduce deployment risk for a high-traffic or business-critical application
• User is debugging a failed deployment and needs to understand rollback procedures and blast radius containment
• User is designing a CI/CD pipeline and needs to choose a deployment pattern that matches their SLO (e.g., <0.1% error rate, <200ms p99 latency)
• User wants to implement progressive delivery -- feature flags, canary releases, or traffic splitting with automated promotion criteria
• User needs to understand the infrastructure prerequisites for a specific strategy (e.g., load balancer requirements for blue/green, service mesh for canary)
• User is migrating from a legacy deploy-and-pray approach to a modern, observable deployment workflow

Do NOT use this skill when:

• User needs infrastructure-as-code authoring (Terraform, Pulumi) -- use the infrastructure-as-code skill
• User needs container orchestration configuration (Kubernetes manifests, Helm charts) beyond what is required to implement a deployment strategy -- use the kubernetes-configuration skill
• User is asking about CI pipeline construction (build steps, test stages, artifact management) without a deployment context -- use the ci-pipeline-design skill
• User needs secrets management or vault configuration as a standalone topic -- use the secrets-management skill
• User is asking about database migration strategies as a standalone concern -- use the database-migration skill
• User needs general monitoring or observability setup -- use the observability-stack skill
• User is asking about incident response after a deployment is already failing in production -- use the incident-response skill

Process

1. Characterize the Application and Traffic Profile

Before recommending any strategy, gather these concrete facts:

• Traffic volume: Requests per second (RPS) at peak. A service handling 50 RPS tolerates more disruption than one at 50,000 RPS. Strategies that require traffic draining (blue/green) need 30--120 seconds of overlap capacity.
• Deployment frequency: Teams deploying once a week can tolerate heavier-weight strategies (blue/green with full environment provisioning). Teams deploying 20+ times per day need lightweight, automated strategies (rolling or canary with automated promotion).
• Session stickiness requirements: Stateful applications (WebSocket connections, server-side sessions) cannot survive mid-session instance replacement. Identify whether sessions can be externalized (Redis session store, JWT tokens) before choosing a strategy.
• Database migration coupling: Does this deploy include schema changes? If yes, the deployment strategy must be backward-compatible migration aware. Blue/green cannot be used naively with breaking schema changes -- the old environment will fail if it shares a database with the new schema.
• SLO targets: Note the specific error budget. If SLO is 99.9% availability (8.7 hours downtime/year), even a 2-minute outage consumes 1.4% of the annual budget. This often rules out the Recreate strategy for production.
• Rollback time objective (RTO): Ask: "How quickly must you be able to roll back?" Blue/green achieves <30 seconds. Rolling rollbacks take minutes proportional to replica count. This is the single most important differentiator between strategies.

2. Apply the Deployment Strategy Decision Framework

Use this decision tree in order:

• Step 2a -- Can the environment sustain duplicate infrastructure cost?

  • Yes (cloud, auto-scaling, sufficient budget): Blue/green is viable. Proceed to 2b.
  • No (fixed hardware, cost-constrained, bare metal): Rolling or Canary are the primary options.

• Step 2b -- Is near-instant rollback (<60 seconds) a hard requirement?

  • Yes: Blue/green is the strongest choice. DNS or load balancer flip achieves it.
  • No: Canary or rolling with automated rollback gates provides sufficient safety at lower cost.

• Step 2c -- Is this a user-facing service where you need behavioral validation before full rollout?

  • Yes, and you can instrument user metrics (conversion rate, error rate per segment): Canary with automated promotion criteria.
  • Yes, but behavioral metrics are hard to define: Blue/green with manual validation gate.
  • No (internal service, batch job, worker): Rolling with health check gates.

• Step 2d -- Is downtime acceptable? (Scheduled maintenance window, internal tooling, non-production)

  • Yes: Recreate strategy -- simplest possible. Stop old, start new.
  • No: Eliminate Recreate from consideration.

• Step 2e -- Do you need to test with real production traffic without user impact?

  • Yes: Shadow deployment (traffic mirroring). Run the new version alongside production, mirror real traffic to it, compare responses. Zero user impact, highest fidelity testing.

3. Define Automated Promotion and Rollback Criteria

Every deployment strategy must have explicit, metric-based gates -- not manual approval for routine deploys:

• Error rate gate: HTTP 5xx rate >1% of requests over a 5-minute window triggers automatic rollback. Tune the threshold based on baseline error rate (never tighter than 3x the p99 baseline variance).
• Latency gate: p99 latency increase of >20% relative to the previous 30-minute baseline triggers rollback. Use relative thresholds, not absolute, to account for traffic-dependent variance.
• Saturation gate: CPU >80% sustained for 3 minutes or memory >90% triggers rollback. These indicate the new version may degrade under load.
• Business metric gate (canary-specific): For canary, define a primary business metric: add-to-cart rate, checkout completion, API success rate. A >5% relative degradation compared to the stable cohort triggers automatic rollback.
• Soak time: Define minimum time at each canary stage (e.g., 10% traffic for 10 minutes, 25% for 15 minutes, 50% for 20 minutes) before automatic promotion. Even if metrics look good, statistical significance requires time.
• Canary stages: Do not jump from 0% to 100%. Use stages: 1%, 5%, 10%, 25%, 50%, 100%. Each stage doubles (approximately) exposure while keeping blast radius bounded.

4. Design the Infrastructure Prerequisites

Each strategy has specific infrastructure requirements -- confirm these before implementation:

• Blue/Green requirements:

  • Two identical environment slots (blue and green) -- either pre-provisioned or spin-up-on-deploy
  • A traffic switching mechanism: Load balancer target group swap (ALB, NLB), DNS TTL swap (set TTL to 60 seconds before deploy, not 300), or service mesh weighted routing
  • Shared or replicated data stores -- the two environments must read from the same database, or you must handle data sync
  • Warm-up time budget: New environment needs time to fill JVM caches, warm connection pools, load ML models. Account for 30--300 seconds of warm-up before switching traffic.

• Canary/Rolling requirements:

  • Load balancer or service mesh with weighted traffic routing (NGINX with split_clients, Envoy with weighted clusters, Kubernetes with service mesh like Istio or Linkerd)
  • Observability stack capable of segmenting metrics by version label -- every metric must carry a version or deployment_id tag so canary vs. stable can be compared
  • At least 2x replica headroom during the transition period -- rolling updates temporarily reduce capacity
  • Pod disruption budgets (Kubernetes) or equivalent to prevent rolling update from removing too many instances simultaneously

• Shadow/Traffic Mirroring requirements:

  • Service mesh or proxy with traffic mirroring capability (Envoy, NGINX mirror module, AWS ALB with mirror target)
  • Mirrored traffic must not trigger side effects (no emails, no payments, no database writes in the shadow path) -- use a separate database or mock downstream services
  • Sufficient compute capacity to handle 2x traffic (shadow runs the request twice)

5. Implement the Deployment Pipeline

Translate the chosen strategy into pipeline stages:

• Pre-deploy gate:

  • Run smoke tests against the artifact before deploying to any environment
  • Validate configuration schema (environment variables, feature flags) against a contract
  • Check that the database migration (if any) is backward-compatible with the current deployed version
  • Confirm infrastructure prerequisites are ready (environment slot exists, load balancer is configured)

• Deploy stage (strategy-specific):

  • For rolling: Set maxUnavailable: 0 and maxSurge: 25% (or 1 replica, whichever is larger) to avoid capacity reduction during rollout. This ensures you never have fewer running instances than baseline.
  • For blue/green: Deploy the green environment, run health checks and integration tests against the green environment directly (via internal DNS or direct IP), then switch traffic only after validation passes.
  • For canary: Deploy the canary Deployment with a small replica count and use weighted service routing. Do NOT use Kubernetes Deployment replicas alone for canary traffic splitting -- replica ratio is an unreliable proxy for traffic percentage. Use a proper traffic management layer.

• Health check validation:

  • Readiness probe must pass on all new instances before traffic is shifted. Configure readiness probes with initialDelaySeconds: 10, periodSeconds: 5, failureThreshold: 3 as a baseline -- adjust for application startup time.
  • Liveness probe should be distinct from readiness probe. Readiness checks if the instance can serve traffic; liveness checks if the process is alive and should not be restarted.
  • Deep health check endpoint (e.g., /health/ready) should verify database connectivity, cache availability, and downstream service reachability -- not just HTTP 200.

• Traffic shift and observation window:

  • Shift traffic according to the stage plan (canary stages or single flip for blue/green)
  • Observe for the defined soak time at each stage
  • Query the observability stack programmatically -- automated gates should read from Prometheus, Datadog, or CloudWatch metrics APIs, not human eyeballs

• Post-deploy validation:

  • Run synthetic transactions (canary probes) against the new version in production -- actual HTTP requests to business-critical endpoints
  • Verify that distributed tracing shows the new version handling requests (confirms traffic routing is correct)
  • Check that no alerts have fired in the first 15 minutes post-deploy

6. Implement Rollback Procedures

Rollback must be defined before deploy begins, not after something goes wrong:

• Blue/green rollback: Flip the load balancer target group back to blue. Execution time: <30 seconds. This is the "undo button" -- it requires no artifact rebuild, no re-deploy.
• Rolling rollback: kubectl rollout undo deployment/<name> or pipeline trigger that re-deploys the previous artifact tag. Time proportional to replica count -- 10 replicas at 2 replicas/minute = 5 minutes.
• Canary rollback: Remove the canary Deployment and route 100% traffic back to stable. If using Argo Rollouts, kubectl argo rollouts abort <name>.
• Database rollback coupling: If the deploy included a migration, and the migration is not backward-compatible, the database CANNOT be rolled back alongside the application without data loss. This is the most common reason zero-downtime deployments fail. Enforce the expand-contract pattern: never deploy a breaking schema change and application change simultaneously.
• Artifact immutability: Tag all artifacts with immutable identifiers (Git SHA, not latest). latest tags make rollback ambiguous and unreliable.

7. Instrument and Close the Feedback Loop

A deployment strategy without observability is a guess:

• DORA metrics: Track Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Recovery. These four metrics objectively measure deployment strategy effectiveness. Target elite performance: deploy frequency daily or more, lead time <1 day, change failure rate <5%, MTTR <1 hour.
• Deployment annotations: Push a deployment event annotation to your metrics platform every time a deploy occurs. This allows instant visual correlation between deploys and metric changes in dashboards.
• Version tagging: Every metric emitted by your application must carry a version label. This enables side-by-side comparison of canary vs. stable error rates in Prometheus or Datadog.
• Runbook linkage: Every automated alert must link to a runbook that explains how to respond. Deployment-related alerts (e.g., "canary error rate elevated") must include the rollback command in the runbook body.
• Post-deploy review cadence: After every deployment incident (not just P1s), conduct a 30-minute blameless postmortem. Track whether the deployment strategy's automated gates caught the problem or missed it. Use this to tune gate thresholds.

Output Format

When advising on or designing a deployment strategy, produce the following artifacts:

Strategy Selection Summary

Application:         [service name]
Team size:           [N engineers]
Deploy frequency:    [N per day/week]
Traffic profile:     [peak RPS, p99 latency SLO]
Availability SLO:    [99.X%]
Rollback RTO:        [< N seconds/minutes]

Recommended strategy:  [Blue/Green | Canary | Rolling | Recreate | Shadow]
Rationale:             [2-3 sentences citing the specific factors above]
Ruled out:             [Strategy X -- reason; Strategy Y -- reason]

Deployment Gate Configuration

# Automated promotion/rollback criteria
gates:
  error_rate:
    threshold: 1.0%          # HTTP 5xx / total requests
    window: 5m
    action: rollback

  latency_p99:
    threshold: +20%           # relative increase vs. 30m baseline
    window: 5m
    action: rollback

  cpu_saturation:
    threshold: 80%
    sustained_for: 3m
    action: rollback

  soak_time_minimum:
    per_stage: 10m            # minimum time at each canary stage before promotion
    action: hold_until_elapsed

canary_stages:
  - weight: 1
    soak: 10m
  - weight: 5
    soak: 10m
  - weight: 10
    soak: 15m
  - weight: 25
    soak: 15m
  - weight: 50
    soak: 20m
  - weight: 100
    soak: 0m    # final promotion, monitoring continues

Rolling Update Configuration (Kubernetes example)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: [service-name]
  annotations:
    deployment.kubernetes.io/change-cause: "[git-sha] [ticket-id] [brief description]"
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0     # never reduce below baseline capacity
      maxSurge: 2           # allow 2 extra replicas during rollout (33% surge)
  minReadySeconds: 30       # instance must pass readiness for 30s before counted as available
  template:
    spec:
      containers:
        - name: [service-name]
          image: [registry]/[service]:[git-sha]   # NEVER use :latest
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 5
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
      terminationGracePeriodSeconds: 60   # allow in-flight requests to complete

Blue/Green Pipeline Pseudocode

PIPELINE blue_green_deploy(artifact_tag):

  1. PROVISION green environment
     - Clone blue infrastructure (IaC parameterized by slot name)
     - Apply artifact_tag to green deployment
     - Wait for all green replicas to pass readiness probes

  2. VALIDATE green (internal traffic only)
     - Run integration test suite against green internal endpoint
     - Run synthetic transactions against green
     - Assert: error rate < 0.5%, p99 latency < SLO threshold
     IF validation fails:
       DESTROY green environment
       FAIL pipeline with validation report

  3. SHIFT traffic
     - Update load balancer: 100% -> green
     - Record rollback point: blue slot, load balancer previous config

  4. OBSERVE (15 minute soak)
     - Poll error rate, latency, saturation every 30 seconds
     IF any gate breached:
       ROLLBACK: update load balancer 100% -> blue
       ALERT on-call with rollback reason and metrics
       FAIL pipeline

  5. PROMOTE
     - Keep blue environment for 2 hours (fast rollback window)
     - After 2 hours: DESTROY blue or re-tag as next deploy slot
     - ANNOTATE metrics platform with deploy event

  6. POST-DEPLOY
     - Verify synthetic transactions passing
     - Confirm no alerts fired
     - Mark deployment as successful in deployment tracking system

Decision Matrix

Rules

1. NEVER use :latest as an artifact tag in any deployment. Latest is mutable -- you cannot reliably roll back to it because the tag may point to a different image by the time you attempt rollback. Always tag with the immutable Git SHA (12 characters minimum).

2. NEVER deploy a breaking database schema change in the same deployment as the application code that depends on it. Use the expand-contract pattern: Phase 1 -- add the new column (backward-compatible), deploy the app that can read both schemas. Phase 2 -- remove the old column after the new app version is fully deployed and validated.

3. NEVER configure maxUnavailable > 0 in a rolling update for a service with fewer than 4 replicas. With 2 replicas and maxUnavailable: 1, you are running at 50% capacity during the rollout. Set maxUnavailable: 0 and maxSurge: 1 for small replica counts.

4. NEVER set DNS TTL to 300 seconds and use DNS-based blue/green switching. At TTL 300, it takes up to 5 minutes for all clients to see the switch, and rollback is equally slow. Drop TTL to 60 seconds at least 10 minutes before the deploy window.

5. NEVER treat canary rollout by replica count as equivalent to canary rollout by traffic percentage. Two canary pods out of 10 total does not guarantee 20% of traffic goes to canary -- it depends on load balancer behavior, session affinity, and request distribution. Use explicit traffic weighting in your load balancer or service mesh.

6. ALWAYS define rollback criteria and the rollback command before the deployment pipeline runs. The rollback procedure must be documented, tested, and accessible to the on-call engineer in under 60 seconds. Never improvise rollback steps during an incident.

7. ALWAYS ensure the readiness probe endpoint performs a deep health check. A shallow readiness probe (HTTP 200 on /) that does not verify database connectivity will mark an instance as ready before it can actually serve traffic successfully, invalidating your deployment health gates.

8. NEVER skip the minimum soak time gate even when metrics look excellent. Statistical significance requires time. A canary error rate of 0% after 30 seconds on 1% traffic means nothing -- it may have received fewer than 10 requests. Enforce minimum soak times regardless of observed metrics.

9. ALWAYS keep the previous deployment artifact available for at least 2 hours after a successful blue/green promotion. Subtle bugs (memory leaks, cache poisoning, background job failures) often manifest 30--90 minutes after deployment. Keeping the blue environment live enables instant rollback without a rebuild.

10. NEVER instrument deployment success solely by exit code of the deploy command. A Kubernetes kubectl apply returns 0 even if pods are crash-looping. Success must be confirmed by: all replicas passing readiness probes, error rate within bounds, and at least one successful synthetic transaction against the deployed version in production.

Edge Cases

Stateful Applications and Session Affinity

Scenario: The application maintains server-side state (WebSocket connections, long-polling, sticky sessions) and cannot be dropped mid-session during a rolling update.

Handling:

• Configure load balancer sticky sessions (cookie-based affinity, not IP-based -- IP affinity breaks under NAT) before initiating the rollout.
• Set terminationGracePeriodSeconds to at least the maximum expected session duration (for WebSockets, this may be 300--900 seconds).
• Configure the application to reject new connections gracefully during shutdown (SIGTERM handler closes the accept loop, drains existing connections, then exits).
• For rolling updates, maxUnavailable: 0 is mandatory -- never pull an instance while it may be holding active sessions.
• If session duration exceeds a reasonable termination grace period, externalize session state (Redis Cluster with TTL) before attempting any live rolling update strategy. This is a prerequisite, not an optimization.

Database Migration Coupled to Deployment

Scenario: A new application version requires schema changes (column rename, table restructure, index addition) that are incompatible with the running version.

Handling:

• Apply the expand-contract (also called parallel change) pattern across three sequential deployments:
  1. Expand: Add new schema elements (new column, new table) without removing old ones. Deploy application version that writes to both old and new schema simultaneously.
  2. Migrate: Run data migration to backfill new schema from old data. Application continues writing to both. Validate backfill completeness.
  3. Contract: Deploy application version that reads only new schema. Remove old schema elements.
• Each phase must be independently deployable and rollback-safe.
• For blue/green with a shared database: both blue and green must be compatible with the deployed schema simultaneously, since both may be serving traffic during validation.
• Never use ON DELETE CASCADE or destructive migrations in the expand phase -- they cannot be rolled back.

High-Frequency Deployment (>20 Deploys Per Day)

Scenario: A team is deploying dozens of times per day, and heavyweight strategies (full blue/green environment provisioning) introduce unacceptable pipeline latency.

Handling:

• Use pre-warmed environment pools (one blue, one green always running) rather than on-demand provisioning. Provisioning a new environment from scratch takes 3--10 minutes; swapping a pre-warmed slot takes 30 seconds.
• Implement progressive delivery with feature flags as the primary risk-reduction mechanism, separate from the deployment event. The deployment ships the code (dark launch); the feature flag controls who sees it. This decouples deployment risk from feature risk.
• Use rolling updates with aggressive automated gates rather than blue/green for most services. Reserve blue/green for the highest-criticality services (payment processing, authentication) where rollback speed justifies 2x infrastructure cost.
• Cap pipeline cycle time: if a deploy pipeline takes >15 minutes, engineers will batch changes to reduce the overhead, which increases blast radius. Optimize pipeline stages (parallel test execution, layer-cached container builds) to stay under 10 minutes gate-to-production.

Multi-Region and Multi-Cluster Deployment

Scenario: The service runs across 3+ AWS regions or Kubernetes clusters, and a deployment must be coordinated across all of them.

Handling:

• Never deploy to all regions simultaneously. Use a progressive regional rollout: deploy to the lowest-traffic region first (often us-west-2 or a non-primary region), observe for 15--30 minutes, then proceed to additional regions.
• The primary (highest traffic) region should be the last region deployed. This limits blast radius to a fraction of global traffic during the most dangerous phase.
• Define a "region health gate" -- before promoting to the next region, confirm that the previous region's error rate, latency, and saturation are within bounds.
• Implement global rollback as a single operation via traffic weighting at the CDN or global load balancer layer (Cloudflare, AWS Global Accelerator). Do not rely on per-region rollbacks in a multi-region incident -- it takes too long.
• Version skew: during the rollout window, different regions run different versions. Ensure inter-service APIs are backward-compatible for at least 2 consecutive versions. Use semantic versioning for internal APIs and enforce compatibility contracts in CI.

Rollback Fails Due to Database State

Scenario: Automated rollback triggers during a canary deploy, but the new version has already written data in a format the old version cannot read.

Handling:

• This scenario indicates the expand-contract pattern was not followed. The root fix is procedural (enforce backward-compatible migrations), but the immediate handling is:
  1. Do NOT roll back the application if doing so would cause data corruption or application crashes in the old version.
  2. Instead, roll forward with a hotfix that restores read compatibility.
  3. Define this as an incident: the deploy gate failed to catch a data compatibility issue before rollback became unsafe.
• Add a pre-deploy check to the pipeline: query the running database schema version and assert compatibility with both the current and the new application version before the deploy proceeds.
• In the post-incident review, enforce the requirement that all schema migrations be reviewed by a database engineer and tested in a staging environment with production-scale data before merging.

Shadow Deployment Side Effects

Scenario: Shadow/mirroring deployment is used to test a new service version, but the shadow path triggers unintended side effects (emails sent, payments charged, messages published to queues).

Handling:

• Shadow deployments require a strict read-only contract for the mirrored path. The shadow service must never write to production databases, call external payment processors, send notifications, or publish to production message queues.
• Implement a shadow-mode flag in the application that, when enabled, replaces all write operations with no-ops or writes to a shadow-specific database.
• Alternatively, provide stub implementations of all side-effecting dependencies for the shadow environment: a stub payment service that accepts but does not process, a stub email service that logs but does not send.
• Verify the stub/no-op behavior in integration tests before enabling shadow traffic. A single missed write path in shadow mode can result in double-charged customers or duplicate notifications.
• Review shadow response body differences against production -- mismatches in response structure (not just latency/error rates) often reveal bugs that would have caused errors in production.

Canary Deployment on a Service with Low Traffic Volume

Scenario: A service handles 20 RPS peak. At 5% canary weight, only 1 request per second goes to the canary. Statistical significance for a 1% error rate requires hundreds of requests, meaning the soak time must be very long.

Handling:

• Adjust canary stages for low-traffic services: skip the 1% and 5% stages (too little traffic to be meaningful) and start at 10--20%.
• Extend soak times proportionally. For a service at 2 RPS canary traffic, achieving 500 requests requires 250 seconds (4.2 minutes). Set minimum soak time to at least 10 minutes at 10% to accumulate sufficient data.
• Use wider error rate confidence intervals for low-traffic gates. A single 500 error in 50 requests is 2% -- this should not trigger rollback on its own. Require at least 5 errors in the window, or use a Bayesian error rate model rather than a simple threshold.
• Consider using synthetic load (load test against canary) in addition to organic traffic to accelerate validation, especially for performance regression checks.

Example

Input: "We're running a payment processing API on Kubernetes (6 replicas). It handles about 800 RPS at peak, our SLO is 99.95% availability, and we deploy roughly 3--5 times per week. We've been doing rolling updates but had two incidents this year where a bad deploy took 8 minutes to roll back and degraded the service during that time. What deployment strategy should we use and how do we implement it?"

Output:

Analysis

The current situation reveals a specific problem: rolling update rollback takes 8 minutes, and during that window, degraded pods are serving traffic. For a payment API at 99.95% SLO (4.4 hours downtime budget/year), an 8-minute degradation event consumes 3% of the annual budget in a single incident.

The root cause with rolling rollbacks is that kubectl rollout undo re-deploys the previous image using the same rolling update mechanism -- it does not instantly revert traffic. With 6 replicas, even with maxSurge: 2, replacing all pods takes approximately 6--10 minutes.

Recommendation: Blue/Green for this service.

Ruled out:

• Rolling update -- current strategy. Rollback RTO of 6--10 minutes is incompatible with payment SLO. Ruled out.
• Canary -- suitable as a secondary risk gate, but canary rollback still takes 2--5 minutes. Payment processing needs faster rollback as the primary mechanism.
• Recreate -- requires full downtime. Payment API cannot have planned downtime. Ruled out completely.

Strategy Selection Summary

Application:         payment-api
Team size:           ~6 engineers (inferred)
Deploy frequency:    3--5 per week
Traffic profile:     800 RPS peak
Availability SLO:    99.95% (4.4 hours/year)
Rollback RTO:        <60 seconds (required for payment criticality)

Recommended strategy:  Blue/Green (primary) + Canary gates (pre-switch validation)
Rationale:             Blue/green achieves sub-30-second rollback via load balancer
                       flip, which is the only mechanism meeting the 99.95% SLO at
                       800 RPS. Canary gates validate the green environment with 5%
                       of traffic before the full switch, catching regressions before
                       100% exposure.

Ruled out:
  Rolling update  -- 6--10 min rollback violates SLO budget
  Canary (only)   -- 2--5 min rollback still too slow for payment path
  Recreate        -- requires downtime, never appropriate for payment APIs

Infrastructure Changes Required

The current rolling update setup needs these additions:

1. Two persistent environment slots -- Maintain a "blue" and "green" Deployment in the same Kubernetes namespace, or use two separate namespaces. The load balancer (Kubernetes Service + Ingress, or an ALB) routes traffic to the active slot.

2. Traffic switching mechanism -- Use Kubernetes Service selector labels to switch traffic. The Service selects pods by label slot: blue or slot: green. Switching is instantaneous: one kubectl patch service payment-api -p '{"spec":{"selector":{"slot":"green"}}}' command.

3. Internal validation endpoint -- The green Deployment needs a separate internal Service (not exposed externally) for running integration tests against it before the traffic switch.

Deployment Gate Configuration

# payment-api deployment gates
gates:
  error_rate:
    metric: 'rate(http_requests_total{status=~"5..",service="payment-api"}[5m]) /
             rate(http_requests_total{service="payment-api"}[5m])'
    threshold: 0.5%        # tighter than default for payment -- half of 1%
    window: 5m
    action: rollback

  latency_p99:
    metric: 'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket
             {service="payment-api"}[5m]))'
    threshold: +15%        # relative increase vs. 30m baseline
    window: 5m
    action: rollback

  payment_success_rate:    # business metric gate -- specific to payment API
    metric: 'rate(payment_transactions_total{status="success"}[5m]) /
             rate(payment_transactions_total[5m])'
    threshold: -2%         # alert if success rate drops more than 2 percentage points
    window: 5m
    action: rollback

  soak_time_pre_switch:    # minimum observation at 5% traffic (canary pre-validation)
    duration: 15m
    action: hold_until_elapsed

canary_pre_validation_stages:
  # Run these against green BEFORE full traffic switch
  - weight: 5
    soak: 15m
    gate: all_above
  # If 5% passes for 15 minutes, proceed to full switch
  # Full switch is atomic (load balancer selector flip), not gradual

Kubernetes Configuration

Blue and Green Deployments:

# green deployment (deployed during each release cycle)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api-green
  namespace: payments
  annotations:
    deployment.kubernetes.io/change-cause: "a3f92c1 PYMT-1042 fix retry idempotency"
spec:
  replicas: 6
  selector:
    matchLabels:
      app: payment-api
      slot: green
  strategy:
    type: RollingUpdate    # used only within the green slot during warm-up
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 2
  minReadySeconds: 30
  template:
    metadata:
      labels:
        app: payment-api
        slot: green
        version: "a3f92c1"   # git sha -- enables per-version metric segmentation
    spec:
      containers:
        - name: payment-api
          image: registry.example.com/payment-api:a3f92c1
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health/ready   # checks DB, cache, downstream payment processor
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 5
            failureThreshold: 3
            successThreshold: 2     # must pass TWICE before marked ready (extra safety)
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3
          resources:
            requests:
              cpu: 500m
              memory: 768Mi
            limits:
              cpu: 2000m
              memory: 1.5Gi
          env:
            - name: SLOT
              value: "green"
            - name: VERSION
              value: "a3f92c1"
      terminationGracePeriodSeconds: 90    # payment requests can take up to 60s
      podDisruptionBudget:                 # defined separately, shown below
        maxUnavailable: 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api-pdb
  namespace: payments
spec:
  maxUnavailable: 1          # at most 1 pod unavailable at a time (cluster maintenance)
  selector:
    matchLabels:
      app: payment-api
      slot: blue             # PDB applies to whichever slot is active

Service (traffic switching mechanism):

apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments
spec:
  selector:
    app: payment-api
    slot: blue              # CURRENT ACTIVE SLOT -- change to "green" to switch traffic
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP
---
# Internal validation service -- routes to green without affecting production traffic
apiVersion: v1
kind: Service
metadata:
  name: payment-api-green-internal
  namespace: payments
spec:
  selector:
    app: payment-api
    slot: green
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

Pipeline Steps

PIPELINE payment-api-blue-green-deploy(git_sha):

PRE-CHECKS (fail fast):
  - Assert image registry.example.com/payment-api:[git_sha] exists and is signed
  - Run db-migration-compatibility-check: assert migration is backward-compatible
    with current running version
  - Assert green slot Deployment does not already exist (clean slate)

STEP 1 -- Deploy green (5 min):
  - Apply payment-api-green Deployment with image tag [git_sha]
  - Wait: all 6 replicas pass readiness probes
  - If timeout (10 min): destroy green, fail pipeline, page on-call

STEP 2 -- Internal integration tests against green (3 min):
  - POST /payments/initiate against payment-api-green-internal Service
  - Assert: 200 response, idempotency key handled correctly
  - Assert: database write confirmed (read-back)
  - Assert: no errors in green pod logs in last 60 seconds
  - If any assertion fails: destroy green, fail pipeline

STEP 3 -- Canary pre-validation (15 min soak at 5%):
  - Update Ingress: route 5% of traffic to green (Nginx split_clients or NGINX Plus)
  - Start 15-minute observation timer
  - Poll every 30s: error_rate, latency_p99, payment_success_rate
  - If any gate breaches: route 100% back to blue, destroy green, fail pipeline
  - If soak completes successfully: proceed

STEP 4 -- Full traffic switch (<30 seconds):
  - kubectl patch service payment-api -n payments \
      -p '{"spec":{"selector":{"slot":"green"}}}'
  - Record rollback command: kubectl patch service payment-api -n payments \
      -p '{"spec":{"selector":{"slot":"blue"}}}'
  - Annotate Grafana, Datadog with deployment event: service=payment-api,
    version=[git_sha], slot=green

STEP 5 -- Post-switch observation (15 min):
  - Run synthetic transaction every 30 seconds: full payment flow end-to-end
  - Poll all gates as in Step 3
  - If any gate breaches: IMMEDIATE rollback (selector flip to blue), page on-call
  - Rollback completes in <30 seconds

STEP 6 -- Promote and cleanup:
  - Mark deploy as successful in deployment tracking
  - Retain blue Deployment for 2 hours (fast rollback window)
  - After 2 hours, if no rollback: delete blue Deployment, tag blue slot as next
    deploy target
  - Update PodDisruptionBudget selector to "green"

STEP 7 -- Post-deploy validation:
  - Confirm DORA metrics updated: deployment frequency, lead time
  - Verify no new alerts firing in PagerDuty
  - Send deploy notification to #payments-deploys Slack channel with
    rollback command embedded in message body

Expected Improvements

The 25-minute pipeline duration is the primary trade-off. For a payment API deploying 3--5 times per week, 25 minutes is acceptable. If pipeline duration becomes a blocker, the canary pre-validation stage (Step 3) can be parallelized with synthetic load to reduce the soak time while maintaining statistical significance.