Cicd Agent

Skill: CICDAgent (Pipeline Generation & Test Automation)

Purpose

Generate comprehensive CI/CD pipelines that automate building, testing, security scanning, and deployment of applications. Ensures all code changes go through automated quality gates before reaching production.

Core Responsibilities

1. Pipeline Generation: Create platform-specific CI/CD configurations (GitHub Actions, GitLab CI, Azure DevOps, etc.)
2. Test Automation: Configure unit, integration, e2e test execution in pipeline
3. Security Scanning: Integrate SAST, DAST, dependency, and IaC security scans
4. Quality Gates: Enforce coverage thresholds, lint rules, complexity limits
5. Artifact Management: Configure versioning, tagging, registry publishing
6. Deployment Automation: Generate deployment scripts for multiple environments (dev/staging/prod)
7. Evidence Collection: Capture logs, test reports, security scan results for auditing

Inputs (REQUIRED)

• TEST Card: defines test strategy (unit/integration/e2e, coverage targets)
• NFR TEST Card: defines non-functional test requirements (performance, load, security)
• World Model: invariant requirements (INV-020: CI/CD, INV-021: automated testing, INV-042: security testing)
• SPEC Card: application type (web app, API, CLI, library), tech stack, deployment target
• Constraints: CI/CD platform (GitHub, GitLab, Azure DevOps), budget, time

Output: CI/CD Evidence Card

CI/CD Evidence Card Schema

### CI/CD Evidence Card: <Project Name>
- **Pipeline definition location**: [path to .github/workflows/*.yml or .gitlab-ci.yml]
- **CI/CD Platform**: [GitHub Actions | GitLab CI | Azure DevOps | Jenkins | CircleCI]
- **Test stages**:
  - Unit tests: [framework, command, coverage target]
  - Integration tests: [framework, command, required services]
  - E2E tests: [framework, command, browser requirements]
- **Security scans**:
  - SAST (Static): [tool, languages scanned, severity threshold]
  - DAST (Dynamic): [tool, endpoints tested, scan duration]
  - Dependency scanning: [tool, registries checked, vulnerability DB]
  - IaC scanning: [tool, files scanned, policy violations]
- **Quality thresholds**:
  - Code coverage: [target %]
  - Lint pass rate: [100% | configurable]
  - Complexity limits: [cyclomatic complexity < X]
  - Security vulnerabilities: [critical: 0, high: 0, med: <5]
- **Artifact versioning**:
  - Scheme: [semantic versioning 2.0.0]
  - Registry: [Docker Hub | GitHub Packages | npm | PyPI]
  - Tagging strategy: [git tag, commit SHA, build number]
- **Deployment automation**:
  - Environments: [dev, staging, prod]
  - Deployment strategy: [rolling update | blue-green | canary]
  - Rollback capability: [automatic on failure | manual trigger]
- **Evidence required**:
  - Build logs: [stored for 90 days]
  - Test reports: [JUnit XML, coverage reports]
  - Security scan reports: [SARIF format, uploaded to code scanning]
  - Deployment logs: [timestamped, includes version deployed]
- **Failure conditions**:
  - Test failure: [any test fails → block merge]
  - Coverage drop: [coverage decreases by >5% → block merge]
  - Security critical/high: [any critical/high vulnerability → block deploy]
  - Build failure: [compilation error → block merge]

Operating Rules (Non-Negotiable)

1. No Manual Testing

• Rule: Manual-only testing = FAIL
• Rationale: Manual testing doesn't scale, isn't reproducible, lacks audit trail
• Enforcement: Pipeline must automate ALL tests (unit, integration, e2e, security, performance)
• Exception: Manual exploratory testing AFTER automated tests pass (not instead of)

2. Security Scans Mandatory for Production

• Rule: Missing security scans for prod = FAIL
• Rationale: Production code must be scanned for vulnerabilities (INV-042)
• Enforcement:
  • SAST scan (CodeQL, Snyk, SonarQube) on every commit
  • Dependency scan (npm audit, Dependabot, Snyk) daily + on dependency changes
  • DAST scan (OWASP ZAP) on staging environment before prod deploy
  • IaC scan (Checkov, tfsec) on infrastructure changes
• Thresholds: 0 critical, 0 high vulnerabilities to deploy to production

3. Pipelines Must Be Reproducible

• Rule: Same code + same pipeline → same result (deterministic builds)
• Rationale: Debugging requires reproducibility, compliance requires audit trail
• Enforcement:
  • Pin all dependency versions (no latest tags)
  • Use lockfiles (package-lock.json, Pipfile.lock, Cargo.lock)
  • Run in containers (Docker ensures consistent environment)
  • Cache dependencies (but bust cache on lockfile changes)

4. Evidence Must Be Retained

• Rule: All pipeline runs produce auditable evidence (logs, reports, artifacts)
• Rationale: Compliance (INV-029: 7-year audit), debugging, incident response
• Enforcement:
  • Store build logs for 90 days minimum
  • Archive test reports permanently (JUnit XML, coverage badges)
  • Upload security scan results to code scanning dashboard
  • Tag deployed artifacts with commit SHA, build timestamp, deployer

Pipeline Templates by Platform

Template 1: GitHub Actions (Node.js + TypeScript)

Target: Node.js 20 LTS, TypeScript 5.x, Jest tests, ESLint, Prettier

File: .github/workflows/ci.yml (150 lines)

name: CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

env:
  NODE_VERSION: '20.x'

jobs:
  # Job 1: Lint and Format Check
  lint:
    name: Lint & Format Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run ESLint
        run: npm run lint
      
      - name: Run Prettier check
        run: npm run format:check
      
      - name: Run TypeScript type check
        run: npm run type-check

  # Job 2: Unit Tests
  test:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run unit tests with coverage
        run: npm run test:coverage
      
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage/coverage-final.json
          flags: unittests
          fail_ci_if_error: true
      
      - name: Check coverage threshold
        run: |
          COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
          if (( $(echo "$COVERAGE < 80" | bc -l) )); then
            echo "Coverage $COVERAGE% is below 80% threshold"
            exit 1
          fi

  # Job 3: Integration Tests
  integration-test:
    name: Integration Tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: testdb
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
      
      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Run Prisma migrations
        run: npx prisma migrate deploy
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/testdb
      
      - name: Run integration tests
        run: npm run test:integration
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/testdb
          REDIS_URL: redis://localhost:6379

  # Job 4: Security Scanning (SAST)
  security-sast:
    name: Security SAST Scan
    runs-on: ubuntu-latest
    permissions:
      security-events: write
      actions: read
      contents: read
    steps:
      - uses: actions/checkout@v4
      
      - name: Initialize CodeQL
        uses: github/codeql-action/init@v3
        with:
          languages: javascript-typescript
      
      - name: Autobuild
        uses: github/codeql-action/autobuild@v3
      
      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@v3
      
      - name: Run Snyk security scan
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high --fail-on=all

  # Job 5: Dependency Vulnerability Scan
  dependency-scan:
    name: Dependency Vulnerability Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
      
      - name: Run npm audit
        run: npm audit --audit-level=high
      
      - name: Check for outdated dependencies
        run: npm outdated || true

  # Job 6: Build Docker Image
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    needs: [lint, test, integration-test, security-sast, dependency-scan]
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=semver,pattern={{version}}
            type=sha,prefix={{branch}}-
      
      - name: Build and push
        uses: docker build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  # Job 7: Deploy to Staging (on main branch)
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    needs: [build]
    if: github.ref == 'refs/heads/main'
    environment:
      name: staging
      url: https://staging.example.com
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to staging
        run: |
          echo "Deploying to staging environment..."
          # Add deployment commands (e.g., kubectl apply, terraform apply, etc.)

Key Features:

• ✅ 7 jobs: lint, unit tests, integration tests, SAST, dependency scan, build, deploy
• ✅ Parallel execution (lint + test + integration run simultaneously)
• ✅ Service containers (PostgreSQL, Redis for integration tests)
• ✅ CodeQL SAST scan + Snyk scan
• ✅ Coverage enforcement (80% threshold)
• ✅ Docker image build with caching
• ✅ Automated staging deployment on main branch
• ✅ Evidence collection (test reports, coverage, security scan results)

Template 2: GitHub Actions (Python + FastAPI)

Target: Python 3.11+, FastAPI, pytest, mypy, black, ruff

File: .github/workflows/ci.yml (160 lines)

name: CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

env:
  PYTHON_VERSION: '3.11'

jobs:
  # Job 1: Lint and Format Check
  lint:
    name: Lint & Format Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'
      
      - name: Install dependencies
        run: |
          pip install -r requirements-dev.txt
      
      - name: Run ruff linter
        run: ruff check .
      
      - name: Run black format check
        run: black --check .
      
      - name: Run mypy type check
        run: mypy src/

  # Job 2: Unit Tests
  test:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -r requirements-dev.txt
      
      - name: Run pytest with coverage
        run: |
          pytest --cov=src --cov-report=xml --cov-report=html tests/unit
      
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage.xml
          flags: unittests
          fail_ci_if_error: true
      
      - name: Check coverage threshold
        run: |
          coverage report --fail-under=80

  # Job 3: Integration Tests with PostgreSQL
  integration-test:
    name: Integration Tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: testdb
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install -r requirements-dev.txt
      
      - name: Run Alembic migrations
        run: alembic upgrade head
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/testdb
      
      - name: Run integration tests
        run: pytest tests/integration
        env:
          DATABASE_URL: postgresql://postgres:postgres@localhost:5432/testdb

  # Job 4: Security Scanning
  security:
    name: Security Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Run Bandit security linter
        run: |
          pip install bandit[toml]
          bandit -r src/ -ll
      
      - name: Run Safety dependency check
        run: |
          pip install safety
          safety check --json
      
      - name: Run Snyk scan
        uses: snyk/actions/python@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high

  # Job 5: Build Docker Image
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    needs: [lint, test, integration-test, security]
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            ${{ secrets.DOCKERHUB_USERNAME }}/myapp:latest
            ${{ secrets.DOCKERHUB_USERNAME }}/myapp:${{ github.sha }}

Key Features:

• ✅ Python-specific: ruff (linter), black (formatter), mypy (type checker)
• ✅ pytest with coverage (80% threshold)
• ✅ Security: Bandit (SAST), Safety (dependency check), Snyk
• ✅ Alembic migrations for database setup
• ✅ Docker build and push to Docker Hub

Template 3: GitLab CI (Multi-Stage Pipeline)

Target: Universal template for any language/framework

File: .gitlab-ci.yml (120 lines)

stages:
  - lint
  - test
  - security
  - build
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

# Lint Stage
lint:
  stage: lint
  image: node:20-alpine
  script:
    - npm ci
    - npm run lint
    - npm run format:check
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
  only:
    - branches

# Unit Test Stage
test:unit:
  stage: test
  image: node:20-alpine
  script:
    - npm ci
    - npm run test:coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
  only:
    - branches

# Integration Test Stage
test:integration:
  stage: test
  image: node:20-alpine
  services:
    - postgres:16-alpine
    - redis:7-alpine
  variables:
    POSTGRES_DB: testdb
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: postgres
    DATABASE_URL: postgresql://postgres:postgres@postgres:5432/testdb
    REDIS_URL: redis://redis:6379
  script:
    - npm ci
    - npx prisma migrate deploy
    - npm run test:integration
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
  only:
    - branches

# Security SAST Scan
security:sast:
  stage: security
  image: returntocorp/semgrep
  script:
    - semgrep --config=auto --junit-xml -o semgrep-report.xml
  artifacts:
    reports:
      junit: semgrep-report.xml
  only:
    - branches

# Dependency Vulnerability Scan
security:dependencies:
  stage: security
  image: node:20-alpine
  script:
    - npm audit --audit-level=high
  allow_failure: true
  only:
    - branches

# Build Docker Image
build:
  stage: build
  image: docker:24-dind
  services:
    - docker:24-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -t $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    - main

# Deploy to Staging
deploy:staging:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
  script:
    - echo "Deploying to staging..."
    - curl -X POST $STAGING_DEPLOY_WEBHOOK
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - main

# Deploy to Production (manual)
deploy:production:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
  script:
    - echo "Deploying to production..."
    - curl -X POST $PRODUCTION_DEPLOY_WEBHOOK
  environment:
    name: production
    url: https://example.com
  when: manual
  only:
    - main

Key Features:

• ✅ 5 stages: lint, test, security, build, deploy
• ✅ Parallel test execution (unit + integration)
• ✅ Service containers (PostgreSQL, Redis)
• ✅ Semgrep SAST scan
• ✅ Coverage reporting (Cobertura format)
• ✅ Docker build with multi-tag (commit SHA + latest)
• ✅ Staging auto-deploy, production manual-deploy
• ✅ Artifact retention (test reports, coverage)

Template 4: Azure DevOps Pipeline (Enterprise)

Target: Enterprise .NET applications with Azure deployment

File: azure-pipelines.yml (180 lines)

trigger:
  branches:
    include:
      - main
      - develop

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'
  dotnetVersion: '8.0.x'
  azureSubscription: 'MyAzureSubscription'
  appServiceName: 'myapp-prod'

stages:
- stage: Build
  displayName: 'Build and Test'
  jobs:
  - job: BuildJob
    displayName: 'Build .NET Application'
    steps:
    - task: UseDotNet@2
      displayName: 'Install .NET SDK'
      inputs:
        version: $(dotnetVersion)
        packageType: sdk
    
    - task: DotNetCoreCLI@2
      displayName: 'Restore NuGet packages'
      inputs:
        command: 'restore'
        projects: '**/*.csproj'
    
    - task: DotNetCoreCLI@2
      displayName: 'Build solution'
      inputs:
        command: 'build'
        projects: '**/*.csproj'
        arguments: '--configuration $(buildConfiguration) --no-restore'
    
    - task: DotNetCoreCLI@2
      displayName: 'Run unit tests'
      inputs:
        command: 'test'
        projects: '**/*Tests.csproj'
        arguments: '--configuration $(buildConfiguration) --no-build --collect:"XPlat Code Coverage" --logger trx'
        publishTestResults: true
    
    - task: PublishCodeCoverageResults@1
      displayName: 'Publish code coverage'
      inputs:
        codeCoverageTool: 'Cobertura'
        summaryFileLocation: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
        failIfCoverageEmpty: true
    
    - task: DotNetCoreCLI@2
      displayName: 'Publish application'
      inputs:
        command: 'publish'
        publishWebProjects: true
        arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'
        zipAfterPublish: true
    
    - task: PublishBuildArtifacts@1
      displayName: 'Publish artifacts'
      inputs:
        PathtoPublish: '$(Build.ArtifactStagingDirectory)'
        ArtifactName: 'drop'
        publishLocation: 'Container'

- stage: SecurityScan
  displayName: 'Security Scanning'
  dependsOn: Build
  jobs:
  - job: SecurityJob
    displayName: 'Run security scans'
    steps:
    - task: UseDotNet@2
      inputs:
        version: $(dotnetVersion)
    
    - task: DotNetCoreCLI@2
      displayName: 'Run vulnerability scan'
      inputs:
        command: 'custom'
        custom: 'list'
        arguments: 'package --vulnerable --include-transitive'
      continueOnError: true
    
    - task: CredScan@3
      displayName: 'Run Credential Scanner'
      inputs:
        outputFormat: 'sarif'
        debugMode: false
    
    - task: Semmle@1
      displayName: 'Run CodeQL Analysis'
      inputs:
        sourceCodeDirectory: '$(Build.SourcesDirectory)'
        language: 'csharp'
        querySuite: 'security-extended'
    
    - task: PublishSecurityAnalysisLogs@3
      displayName: 'Publish security logs'
      inputs:
        ArtifactName: 'SecurityLogs'
        ArtifactType: 'Container'

- stage: DeployStaging
  displayName: 'Deploy to Staging'
  dependsOn: SecurityScan
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: DeployStaging
    displayName: 'Deploy to Azure App Service (Staging)'
    environment: 'staging'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadBuildArtifacts@1
            inputs:
              buildType: 'current'
              downloadType: 'single'
              artifactName: 'drop'
              downloadPath: '$(System.ArtifactsDirectory)'
          
          - task: AzureWebApp@1
            displayName: 'Deploy to staging slot'
            inputs:
              azureSubscription: $(azureSubscription)
              appType: 'webApp'
              appName: $(appServiceName)
              package: '$(System.ArtifactsDirectory)/drop/**/*.zip'
              deploymentMethod: 'auto'
              slotName: 'staging'

- stage: DeployProduction
  displayName: 'Deploy to Production'
  dependsOn: DeployStaging
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: DeployProduction
    displayName: 'Deploy to Azure App Service (Production)'
    environment: 'production'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: DownloadBuildArtifacts@1
            inputs:
              buildType: 'current'
              downloadType: 'single'
              artifactName: 'drop'
              downloadPath: '$(System.ArtifactsDirectory)'
          
          - task: AzureWebApp@1
            displayName: 'Deploy to production'
            inputs:
              azureSubscription: $(azureSubscription)
              appType: 'webApp'
              appName: $(appServiceName)
              package: '$(System.ArtifactsDirectory)/drop/**/*.zip'
              deploymentMethod: 'auto'

Key Features:

• ✅ 4 stages: Build, SecurityScan, DeployStaging, DeployProduction
• ✅ .NET 8.0 SDK with NuGet restore
• ✅ Unit tests with code coverage (Cobertura format)
• ✅ Security scans: vulnerability scan, CredScan, CodeQL
• ✅ Azure App Service deployment with staging slot
• ✅ Environment-based deployments (staging auto, production requires approval)
• ✅ Artifact publishing with versioning

Deployment Strategies

Strategy 1: Rolling Update (Zero Downtime)

Use Case: Standard deployments for stateless applications

How It Works:

1. Deploy new version to subset of instances (e.g., 25%)
2. Wait for health checks to pass
3. Continue rolling out to remaining instances in batches
4. Old instances terminated only after new instances are healthy

Implementation (Kubernetes):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1  # Only 1 pod down at a time
      maxSurge: 1        # Only 1 extra pod during update
  template:
    spec:
      containers:
      - name: myapp
        image: myapp:v2.0.0
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

Advantages:

• ✅ Zero downtime
• ✅ Gradual rollout reduces risk
• ✅ Automatic rollback if health checks fail

Disadvantages:

• ❌ Both old and new versions run simultaneously (version coexistence)
• ❌ Database migrations must be backward compatible

Strategy 2: Blue-Green Deployment (Instant Switchover)

Use Case: Mission-critical applications requiring instant rollback capability

How It Works:

1. Deploy new version (Green) alongside existing version (Blue)
2. Run smoke tests on Green environment
3. Switch traffic from Blue → Green (instant cutover)
4. Keep Blue environment running for 24-48h as rollback option
5. Decommission Blue after validation period

Implementation (GitHub Actions + AWS):

deploy-blue-green:
  runs-on: ubuntu-latest
  steps:
    - name: Deploy to Green environment
      run: |
        aws ecs update-service \
          --cluster production \
          --service myapp-green \
          --task-definition myapp:${{ github.sha }} \
          --desired-count 4
    
    - name: Wait for Green to be healthy
      run: |
        aws ecs wait services-stable \
          --cluster production \
          --services myapp-green
    
    - name: Run smoke tests on Green
      run: |
        curl -f https://green.myapp.com/health || exit 1
        curl -f https://green.myapp.com/api/status || exit 1
    
    - name: Switch traffic to Green
      run: |
        aws elbv2 modify-listener \
          --listener-arn ${{ secrets.ALB_LISTENER_ARN }} \
          --default-actions \
            Type=forward,TargetGroupArn=${{ secrets.GREEN_TARGET_GROUP }}
    
    - name: Monitor for 10 minutes
      run: |
        sleep 600
        ERROR_RATE=$(aws cloudwatch get-metric-statistics \
          --namespace MyApp --metric-name ErrorRate \
          --start-time $(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S) \
          --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
          --period 600 --statistics Average --query 'Datapoints[0].Average')
        
        if (( $(echo "$ERROR_RATE > 5.0" | bc -l) )); then
          echo "High error rate detected: $ERROR_RATE%"
          exit 1
        fi
    
    - name: Scale down Blue (keep as rollback)
      run: |
        aws ecs update-service \
          --cluster production \
          --service myapp-blue \
          --desired-count 1

Advantages:

• ✅ Instant rollback (switch traffic back to Blue)
• ✅ Full production testing before cutover
• ✅ No version coexistence issues

Disadvantages:

• ❌ Requires 2x infrastructure (expensive)
• ❌ Database updates tricky (both environments access same DB)

Strategy 3: Canary Deployment (Gradual Traffic Shift)

Use Case: High-risk changes, new features, performance-sensitive applications

How It Works:

1. Deploy new version to small subset (canary: 5% traffic)
2. Monitor metrics (error rate, latency, business KPIs) for 30-60 minutes
3. If metrics healthy, increase traffic gradually (5% → 25% → 50% → 100%)
4. Roll back immediately if metrics degrade

Implementation (Istio Service Mesh):

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp.example.com
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: myapp
        subset: v2
      weight: 100
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 95  # 95% traffic to stable
    - destination:
        host: myapp
        subset: v2
      weight: 5   # 5% traffic to canary
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

Automation with Flagger (Progressive Delivery):

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 8080
  analysis:
    interval: 1m
    threshold: 5  # Allow 5 failed checks before rollback
    maxWeight: 50
    stepWeight: 10  # Increase by 10% each step
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99  # 99% success rate required
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500  # p99 latency < 500ms
      interval: 1m

Traffic Progression:

• t=0min: 5% canary, 95% stable
• t=1min: 15% canary (if metrics OK)
• t=2min: 25% canary
• t=3min: 35% canary
• t=4min: 45% canary
• t=5min: 50% canary (maxWeight reached)
• Manual promotion to 100% after validation

Advantages:

• ✅ Minimal user impact (only 5% affected if bad)
• ✅ Real production validation before full rollout
• ✅ Automatic rollback based on metrics

Disadvantages:

• ❌ Requires service mesh or feature flags
• ❌ Complex monitoring setup
• ❌ Slower rollout (5-10 minutes vs instant)

Strategy 4: Feature Flags (Dark Launch)

Use Case: Gradual feature rollout, A/B testing, kill switches

How It Works:

1. Deploy new code with feature behind flag (disabled by default)
2. Enable feature for internal users (dogfooding)
3. Enable for 1% of users, then 5%, 10%, 25%, 50%, 100%
4. Monitor feature-specific metrics
5. Disable flag instantly if issues detected (no redeployment needed)

Implementation (LaunchDarkly / Unleash):

// Feature flag check in code
import { LDClient } from 'launchdarkly-node-server-sdk';

const ldClient = LDClient.init(process.env.LAUNCHDARKLY_SDK_KEY);

app.get('/api/products', async (req, res) => {
  const user = {
    key: req.user.id,
    email: req.user.email,
    custom: {
      tenantId: req.user.tenantId,
      userType: req.user.role
    }
  };
  
  // Check if new product recommendation engine enabled
  const useNewRecommendations = await ldClient.variation(
    'product-recommendations-v2',
    user,
    false  // Default to old version if flag unavailable
  );
  
  if (useNewRecommendations) {
    return res.json(await getRecommendationsV2(req.user));
  } else {
    return res.json(await getRecommendationsV1(req.user));
  }
});

Feature Flag Rules (LaunchDarkly Dashboard):

{
  "key": "product-recommendations-v2",
  "variations": [
    {"value": false, "name": "Old Recommendations"},
    {"value": true, "name": "New ML-based Recommendations"}
  ],
  "targeting": {
    "rules": [
      {
        "clauses": [{"attribute": "email", "op": "endsWith", "values": ["@mycompany.com"]}],
        "variation": 1  // Internal users get new version
      },
      {
        "clauses": [{"attribute": "userType", "op": "in", "values": ["beta-tester"]}],
        "variation": 1  // Beta testers get new version
      }
    ],
    "fallthrough": {
      "rollout": {
        "variations": [
          {"variation": 0, "weight": 95000},  // 95% old version
          {"variation": 1, "weight": 5000}    // 5% new version
        ]
      }
    }
  },
  "offVariation": 0  // Kill switch: disable instantly
}

Advantages:

• ✅ Instant enable/disable (no redeployment)
• ✅ Gradual user-based rollout (not infrastructure-based)
• ✅ A/B testing possible (track conversion metrics per variant)
• ✅ Decouple deployment from release

Disadvantages:

• ❌ Code complexity (if/else branches everywhere)
• ❌ Technical debt (old code paths remain until flag cleanup)
• ❌ Testing complexity (must test both code paths)

Rollback Strategies

Rollback Type 1: Revert Git Commit

Trigger: Bad code deployed, bugs in production

Steps:

# Revert last commit on main branch
git revert HEAD
git push origin main

# CI/CD automatically triggers new build + deploy with reverted code

Advantages:

• ✅ Clean git history (revert commit recorded)
• ✅ Automatic via CI/CD (no manual steps)

Disadvantages:

• ❌ Slow (requires full build + test + deploy cycle: 5-15 minutes)
• ❌ Doesn't work for database migrations (can't revert schema changes)

Rollback Type 2: Redeploy Previous Docker Image

Trigger: Current deployment broken, previous version known-good

Steps (Kubernetes):

# Rollback to previous revision
kubectl rollout undo deployment/myapp

# Or rollback to specific revision
kubectl rollout history deployment/myapp
kubectl rollout undo deployment/myapp --to-revision=3

Steps (Docker Compose):

# Switch docker-compose.yml to previous image tag
sed -i 's/myapp:v2.0.0/myapp:v1.9.0/g' docker-compose.yml
docker-compose up -d

Advantages:

• ✅ Fast (2-5 minutes, no rebuild needed)
• ✅ Works even if CI/CD pipeline broken

Disadvantages:

• ❌ Must have previous image in registry (retention policy important)
• ❌ Database migrations not rolled back (schema mismatch risk)

Rollback Type 3: Traffic Switch (Blue-Green)

Trigger: New version deployed but causing errors

Steps:

# Switch ALB back to Blue target group
aws elbv2 modify-listener \
  --listener-arn $LISTENER_ARN \
  --default-actions Type=forward,TargetGroupArn=$BLUE_TARGET_GROUP

# Instant rollback, Green environment remains running for debugging

Advantages:

• ✅ Instant rollback (< 1 second)
• ✅ Green environment available for debugging after rollback
• ✅ Can switch back and forth unlimited times

Disadvantages:

• ❌ Requires Blue-Green infrastructure (2x cost)
• ❌ Database shared between Blue and Green (schema compatibility required)

Rollback Type 4: Feature Flag Disable

Trigger: New feature causing issues

Steps:

1. Open feature flag dashboard (LaunchDarkly, Unleash, etc.)
2. Find flag for problematic feature
3. Set to OFF (disabled)
4. Save (rollout applies instantly to all users)

Advantages:

• ✅ Instant (< 5 seconds, no deployment)
• ✅ Granular (disable specific feature, not entire application)
• ✅ Can re-enable after fix

Disadvantages:

• ❌ Only works for features wrapped in flags
• ❌ Old code path must still work

Evidence Collection Protocols

Evidence Type 1: Build Logs

What: Full output of build process (compilation, linking, packaging)

Collection:

# GitHub Actions
- name: Build application
  run: npm run build
  continue-on-error: true  # Capture logs even on failure

- name: Upload build logs
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: build-logs
    path: build.log
    retention-days: 90

Storage: CI/CD platform artifact storage (GitHub Actions artifacts, GitLab Artifacts, Azure DevOps Artifacts)

Retention: 90 days minimum (compliance: INV-029 requires 7 years for audit logs)

Evidence Type 2: Test Reports

What: Test results (pass/fail), coverage reports, performance benchmarks

Collection (JUnit XML):

- name: Run tests
  run: npm test -- --ci --coverage --reporters=default --reporters=jest-junit

- name: Publish test results
  if: always()
  uses: dorny/test-reporter@v1
  with:
    name: Test Results
    path: junit.xml
    reporter: jest-junit

- name: Upload coverage to Codecov
  uses: codecov/codecov-action@v4
  with:
    files: ./coverage/coverage-final.json
    fail_ci_if_error: true

Formats:

• JUnit XML (test results)
• Cobertura XML (code coverage)
• HTML (human-readable coverage reports)

Retention: Permanent (test history for trend analysis)

Evidence Type 3: Security Scan Reports

What: SAST, DAST, dependency vulnerability scans

Collection (SARIF format):

- name: Run CodeQL analysis
  uses: github/codeql-action/analyze@v3
  with:
    upload: true  # Upload to GitHub Security tab

- name: Run Snyk scan
  run: |
    snyk test --sarif-file-output=snyk.sarif
    snyk monitor  # Send results to Snyk dashboard

- name: Upload SARIF to GitHub
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: snyk.sarif

Formats:

• SARIF (Static Analysis Results Interchange Format)
• JSON (for custom tools)
• HTML (for human review)

Retention: Permanent (audit trail for security compliance)

Evidence Type 4: Deployment Logs

What: Who deployed, when, what version, to which environment

Collection:

- name: Record deployment
  run: |
    echo "Deployment Record" > deployment.log
    echo "Timestamp: $(date -u +%Y-%m-%d %H:%M:%S UTC)" >> deployment.log
    echo "Deployer: ${{ github.actor }}" >> deployment.log
    echo "Commit SHA: ${{ github.sha }}" >> deployment.log
    echo "Image Tag: myapp:$(git describe --tags)" >> deployment.log
    echo "Environment: production" >> deployment.log
    echo "Deployment Method: rolling update" >> deployment.log

- name: Send to audit log system
  run: |
    curl -X POST https://auditlog.company.com/api/deployments \
      -H "Authorization: Bearer ${{ secrets.AUDIT_TOKEN }}" \
      -d @deployment.log

Retention: 7 years (INV-029 compliance requirement)

Failure Analysis & Debugging

Failure Pattern 1: Test Failure

Symptom: CI pipeline fails at test stage

Debugging Steps:

1. Check test report (which tests failed)
2. Review test logs (assertion errors, stack traces)
3. Check if flaky test (passes on retry)
4. Reproduce locally: npm test -- --testNamePattern="failing test name"
5. Fix test or underlying code
6. Commit fix, pipeline re-runs

Common Causes:

• Test depends on external service (mock it)
• Test has race condition (add proper async/await)
• Test assumes specific execution order (make independent)
• Test uses hardcoded timestamp (use mocking library)

Failure Pattern 2: Security Scan Failure

Symptom: SAST or dependency scan finds critical/high

…(truncated)