Cicd Pipeline

CI/CD Pipeline (GitHub Actions)

Set up and manage CI/CD pipelines using GitHub Actions. Covers workflow creation, testing, deployment, release automation, and debugging.

When to Use

• Setting up automated testing on push/PR
• Creating deployment pipelines (staging, production)
• Automating releases with changelogs and tags
• Debugging failing CI workflows
• Setting up matrix builds for cross-platform testing
• Managing secrets and environment variables in CI
• Optimizing CI with caching and parallelism

Quick Start: Add CI to a Project

Node.js project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm test
      - run: npm run lint

Python project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest
      - run: ruff check .

Go project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.22"
      - run: go test ./...
      - run: go vet ./...

Rust project

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo test
      - run: cargo clippy -- -D warnings

Common Patterns

Matrix builds (test across versions/OSes)

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node-version: [18, 20, 22]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

Conditional jobs

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test

  deploy:
    needs: test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh

Caching dependencies

# Node.js (automatic with setup-node)
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm  # or yarn, pnpm

# Generic caching
- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/pip
      ~/.cargo/registry
      node_modules
    key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-deps-

Artifacts (save build outputs)

- uses: actions/upload-artifact@v4
  with:
    name: build-output
    path: dist/
    retention-days: 7

# Download in another job
- uses: actions/download-artifact@v4
  with:
    name: build-output
    path: dist/

Run on schedule (cron)

on:
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6 AM UTC
  workflow_dispatch:  # Also allow manual trigger

Deployment Workflows

Deploy to production on tag

name: Release

on:
  push:
    tags:
      - "v*"

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npm run build
      - run: npm test

      # Create GitHub release
      - uses: softprops/action-gh-release@v2
        with:
          generate_release_notes: true
          files: |
            dist/*.js
            dist/*.css

Deploy to multiple environments

name: Deploy

on:
  push:
    branches: [main, staging]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: |
          if [ "${{ github.ref }}" = "refs/heads/main" ]; then
            ./deploy.sh production
          else
            ./deploy.sh staging
          fi
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

Docker build and push

name: Docker

on:
  push:
    branches: [main]
    tags: ["v*"]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

npm publish on release

name: Publish

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          registry-url: https://registry.npmjs.org
      - run: npm ci
      - run: npm test
      - run: npm publish --provenance
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Secrets Management

Set secrets via CLI

# Set a repository secret
gh secret set DEPLOY_TOKEN --body "my-secret-value"

# Set from a file
gh secret set SSH_KEY < ~/.ssh/deploy_key

# Set for a specific environment
gh secret set DB_PASSWORD --env production --body "p@ssw0rd"

# List secrets
gh secret list

# Delete a secret
gh secret delete OLD_SECRET

Use secrets in workflows

env:
  # Available to all steps in this job
  DATABASE_URL: ${{ secrets.DATABASE_URL }}

steps:
  - run: echo "Deploying..."
    env:
      # Available to this step only
      API_KEY: ${{ secrets.API_KEY }}

Environment protection rules

Set up via GitHub UI or API:

• Required reviewers before deployment
• Wait timers
• Branch restrictions
• Custom deployment branch policies

# View environments
gh api repos/{owner}/{repo}/environments | jq '.environments[].name'

Workflow Debugging

Re-run failed jobs

# List recent workflow runs
gh run list --limit 10

# View a specific run
gh run view <run-id>

# View failed job logs
gh run view <run-id> --log-failed

# Re-run failed jobs only
gh run rerun <run-id> --failed

# Re-run entire workflow
gh run rerun <run-id>

Debug with SSH (using tmate)

# Add this step before the failing step
- uses: mxschmitt/action-tmate@v3
  if: failure()
  with:
    limit-access-to-actor: true

Common failures and fixes

"Permission denied" on scripts

- run: chmod +x ./scripts/deploy.sh && ./scripts/deploy.sh

"Node modules not found"

# Make sure npm ci runs before npm test
- run: npm ci     # Install exact lockfile versions
- run: npm test   # Now node_modules exists

"Resource not accessible by integration"

# Add permissions block
permissions:
  contents: write
  packages: write
  pull-requests: write

Cache not restoring

# Check cache key matches - use hashFiles for lockfile
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
# NOT: key: ${{ runner.os }}-node-${{ hashFiles('package.json') }}

Workflow not triggering

• Check: is the workflow file on the default branch?
• Check: does the trigger event match? (push vs pull_request)
• Check: is the branch filter correct?

# Manually trigger a workflow
gh workflow run ci.yml --ref main

Workflow Validation

Validate locally before pushing

# Check YAML syntax
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" && echo "Valid"

# Use actionlint (if installed)
actionlint .github/workflows/ci.yml

# Or via Docker
docker run --rm -v "$(pwd):/repo" -w /repo rhysd/actionlint:latest

View workflow as graph

# List all workflows
gh workflow list

# View workflow definition
gh workflow view ci.yml

# Watch a running workflow
gh run watch

Advanced Patterns

Reusable workflows

# .github/workflows/reusable-test.yml
name: Reusable Test
on:
  workflow_call:
    inputs:
      node-version:
        required: false
        type: string
        default: "20"
    secrets:
      npm-token:
        required: false

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ inputs.node-version }}
      - run: npm ci
      - run: npm test

# .github/workflows/ci.yml - caller
name: CI
on: [push, pull_request]
jobs:
  test:
    uses: ./.github/workflows/reusable-test.yml
    with:
      node-version: "20"

Concurrency (prevent duplicate runs)

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true  # Cancel previous runs for same branch

Path filters (only run for relevant changes)

on:
  push:
    paths:
      - "src/**"
      - "package.json"
      - "package-lock.json"
      - ".github/workflows/ci.yml"
    paths-ignore:
      - "docs/**"
      - "*.md"

Monorepo: only test changed packages

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'packages/api/**'
            web:
              - 'packages/web/**'

  test-api:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/api && npm ci && npm test

  test-web:
    needs: changes
    if: needs.changes.outputs.web == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd packages/web && npm ci && npm test

Tips

• Use workflow_dispatch on every workflow for manual triggering during debugging
• Pin action versions to SHA for supply chain security: uses: actions/checkout@b4ffde...
• Use continue-on-error: true for non-critical steps (like linting)
• Set timeout-minutes on jobs to prevent runaway builds (default is 360 minutes)
• Use job outputs to pass data between jobs: outputs: result: ${{ steps.step-id.outputs.value }}
• For self-hosted runners: runs-on: self-hosted with labels for targeting specific machines