Deployment Pipeline Design

Deployment Pipeline Design

Architecture patterns for multi-stage CI/CD pipelines with approval gates and deployment strategies.

Purpose

Design robust, secure deployment pipelines that balance speed with safety through proper stage organization and approval workflows.

When to Use

• Design CI/CD architecture
• Implement deployment gates
• Configure multi-environment pipelines
• Establish deployment best practices
• Implement progressive delivery

Pipeline Stages

Standard Pipeline Flow

┌─────────┐   ┌──────┐   ┌─────────┐   ┌────────┐   ┌──────────┐
│  Build  │ → │ Test │ → │ Staging │ → │ Approve│ → │Production│
└─────────┘   └──────┘   └─────────┘   └────────┘   └──────────┘

Detailed Stage Breakdown

1. Source - Code checkout
2. Build - Compile, package, containerize
3. Test - Unit, integration, security scans
4. Staging Deploy - Deploy to staging environment
5. Integration Tests - E2E, smoke tests
6. Approval Gate - Manual approval required
7. Production Deploy - Canary, blue-green, rolling
8. Verification - Health checks, monitoring
9. Rollback - Automated rollback on failure

Approval Gate Patterns

Pattern 1: Manual Approval

# GitHub Actions
production-deploy:
  needs: staging-deploy
  environment:
    name: production
    url: https://app.example.com
  runs-on: ubuntu-latest
  steps:
    - name: Deploy to production
      run: |
        # Deployment commands

Pattern 2: Time-Based Approval

# GitLab CI
deploy:production:
  stage: deploy
  script:
    - deploy.sh production
  environment:
    name: production
  when: delayed
  start_in: 30 minutes
  only:
    - main

Pattern 3: Multi-Approver

# Azure Pipelines
stages:
- stage: Production
  dependsOn: Staging
  jobs:
  - deployment: Deploy
    environment:
      name: production
      resourceType: Kubernetes
    strategy:
      runOnce:
        preDeploy:
          steps:
          - task: ManualValidation@0
            inputs:
              notifyUsers: 'team-leads@example.com'
              instructions: 'Review staging metrics before approving'

Reference: See assets/approval-gate-template.yml

Deployment Strategies

1. Rolling Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1

Characteristics:

• Gradual rollout
• Zero downtime
• Easy rollback
• Best for most applications

2. Blue-Green Deployment

# Blue (current)
kubectl apply -f blue-deployment.yaml
kubectl label service my-app version=blue

# Green (new)
kubectl apply -f green-deployment.yaml
# Test green environment
kubectl label service my-app version=green

# Rollback if needed
kubectl label service my-app version=blue

Characteristics:

• Instant switchover
• Easy rollback
• Doubles infrastructure cost temporarily
• Good for high-risk deployments

3. Canary Deployment

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 25
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 100

Characteristics:

• Gradual traffic shift
• Risk mitigation
• Real user testing
• Requires service mesh or similar

4. Feature Flags

from flagsmith import Flagsmith

flagsmith = Flagsmith(environment_key="API_KEY")

if flagsmith.has_feature("new_checkout_flow"):
    # New code path
    process_checkout_v2()
else:
    # Existing code path
    process_checkout_v1()

Characteristics:

• Deploy without releasing
• A/B testing
• Instant rollback
• Granular control

Pipeline Orchestration

Multi-Stage Pipeline Example

name: Production Pipeline

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build application
        run: make build
      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .
      - name: Push to registry
        run: docker push myapp:${{ github.sha }}

  test:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - name: Unit tests
        run: make test
      - name: Security scan
        run: trivy image myapp:${{ github.sha }}

  deploy-staging:
    needs: test
    runs-on: ubuntu-latest
    environment:
      name: staging
    steps:
      - name: Deploy to staging
        run: kubectl apply -f k8s/staging/

  integration-test:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - name: Run E2E tests
        run: npm run test:e2e

  deploy-production:
    needs: integration-test
    runs-on: ubuntu-latest
    environment:
      name: production
    steps:
      - name: Canary deployment
        run: |
          kubectl apply -f k8s/production/
          kubectl argo rollouts promote my-app

  verify:
    needs: deploy-production
    runs-on: ubuntu-latest
    steps:
      - name: Health check
        run: curl -f https://app.example.com/health
      - name: Notify team
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
            -d '{"text":"Production deployment successful!"}'

Pipeline Best Practices

1. Fail fast - Run quick tests first
2. Parallel execution - Run independent jobs concurrently
3. Caching - Cache dependencies between runs
4. Artifact management - Store build artifacts
5. Environment parity - Keep environments consistent
6. Secrets management - Use secret stores (Vault, etc.)
7. Deployment windows - Schedule deployments appropriately
8. Monitoring integration - Track deployment metrics
9. Rollback automation - Auto-rollback on failures
10. Documentation - Document pipeline stages

Rollback Strategies

Automated Rollback

deploy-and-verify:
  steps:
    - name: Deploy new version
      run: kubectl apply -f k8s/

    - name: Wait for rollout
      run: kubectl rollout status deployment/my-app

    - name: Health check
      id: health
      run: |
        for i in {1..10}; do
          if curl -sf https://app.example.com/health; then
            exit 0
          fi
          sleep 10
        done
        exit 1

    - name: Rollback on failure
      if: failure()
      run: kubectl rollout undo deployment/my-app

Manual Rollback

# List revision history
kubectl rollout history deployment/my-app

# Rollback to previous version
kubectl rollout undo deployment/my-app

# Rollback to specific revision
kubectl rollout undo deployment/my-app --to-revision=3

Monitoring and Metrics

Key Pipeline Metrics

• Deployment Frequency - How often deployments occur
• Lead Time - Time from commit to production
• Change Failure Rate - Percentage of failed deployments
• Mean Time to Recovery (MTTR) - Time to recover from failure
• Pipeline Success Rate - Percentage of successful runs
• Average Pipeline Duration - Time to complete pipeline

Integration with Monitoring

- name: Post-deployment verification
  run: |
    # Wait for metrics stabilization
    sleep 60

    # Check error rate
    ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]')

    if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
      echo "Error rate too high: $ERROR_RATE"
      exit 1
    fi

Reference Files

• references/pipeline-orchestration.md - Complex pipeline patterns
• assets/approval-gate-template.yml - Approval workflow templates

Related Skills

• gitlab-ci-patterns - For GitLab CI implementation
• secrets-management - For secrets handling