CI Pipelines

CI Pipelines

CI in open source has a constraint that internal CI does not: most PRs come from forks, by people who cannot debug your pipeline and will not wait for it. Design for that.

Targets

The last one is the one projects get wrong. If a fork PR always shows a red X because a deploy job cannot access secrets, every contributor's first experience is failure.

Workflow structure

Split by purpose and speed, not into one giant workflow:

.github/workflows/
├── ci.yml            # lint + test on PR and push to main — the required check
├── release.yml       # tag-triggered publish (see release-engineering)
├── codeql.yml        # scheduled security scanning
├── nightly.yml       # slow: full matrix, fuzzing, benchmarks, downstream tests
└── docs.yml          # deploy docs on merge to main

A minimal, correct ci.yml:

name: CI
on:
  push:
    branches: [main]
  pull_request:

# Cancel superseded runs — the single biggest CI cost saving, one block long
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
  contents: read          # least privilege by default; grant per-job as needed

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22', cache: 'npm' }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  test:
    needs: lint                      # don't burn matrix minutes on unformatted code
    strategy:
      fail-fast: false               # show every failing cell, not just the first
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: ['20', '22']
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: ${{ matrix.node }}, cache: 'npm' }
      - run: npm ci
      - run: npm test

Details that matter and are usually missing:

• concurrency with cancel-in-progress — a contributor pushing five times in ten minutes should not queue five full matrices.
• fail-fast: false — otherwise one Windows failure hides three others.
• permissions: contents: read at the top level, elevated only in the job that needs it. Default-permissive tokens are the main blast radius in a compromised action.
• needs: lint — cheap gate first.
• Pin third-party actions to a SHA, not a tag (see supply-chain-security). First-party actions/* by major tag is a defensible compromise; anything else, pin.

Matrix sizing

A full cartesian product is the default and it is almost always wasteful. 3 OS × 4 versions × 2 configs = 24 jobs for a project whose bugs are all on one axis.

Test the corners exhaustively and the middle sparsely:

strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest]
    python: ['3.10', '3.13']            # oldest supported + newest
    include:
      - { os: macos-latest,   python: '3.13' }
      - { os: windows-latest, python: '3.13' }
      - { os: ubuntu-latest,  python: '3.12' }

Move the exhaustive matrix to nightly.yml. PRs get fast signal; the full grid still runs daily and you learn about the rare cell within 24 hours.

Caching

Cache the dependency store, never node_modules/.venv themselves — restoring a partially valid install directory produces confusing failures.

- uses: actions/cache@v4
  with:
    path: ~/.cargo/registry
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
    restore-keys: ${{ runner.os }}-cargo-

Rules:

• Key on the lockfile hash. Keying on anything vaguer serves stale caches.
• Include the OS and language version in the key, or macOS restores Linux binaries.
• restore-keys for partial hits — a near-miss cache still saves most of the time.
• Language-specific setup actions already do this (cache: 'npm', cache: 'pip'). Use them before hand-rolling.
• Never cache anything that affects correctness — build outputs keyed loosely produce "works in CI, broken for users", the worst failure mode available.
• GitHub evicts caches after 7 days of no use, 10 GB per repo. Do not architect around a cache being present.

Fork PRs and secrets

The core security constraint: pull_request from a fork gets a read-only token and no secrets. This is correct and you should not fight it.

• pull_request_target runs with full secrets against the base repo and is the most dangerous trigger in GitHub Actions. If you use it, never check out the PR head — that executes attacker-controlled code with your secrets. Use it only for labeling, commenting, and other metadata work.
• Structure jobs so fork PRs go fully green without secrets. Guard secret-requiring jobs:

  if: github.event.pull_request.head.repo.full_name == github.repository
  

• Coverage upload, preview deploys, and publishing all belong in a separate workflow triggered by workflow_run after CI succeeds, or on push to main.
• Require approval for first-time contributors' workflow runs — repo Settings → Actions. This is the default for new contributors and worth keeping.
• Self-hosted runners on a public repo are a remote code execution service unless they are ephemeral and isolated. Every fork PR can run arbitrary code on them. If you need self-hosted, use ephemeral single-use runners in a disposable VM.

Required checks and branch protection

Configure on main:

• Require status checks to pass, and require branches be up to date only if your merge volume is low — on a busy repo that setting causes an update-rebase treadmill. Prefer a merge queue.
• Require one approving review, dismiss stale approvals on new commits.
• Require conversation resolution.
• Include administrators. If you exempt yourself, the rules are advisory.
• Do not require jobs that cannot run on forks — this is the most common way a repo becomes unmergeable for outside contributors.

Name required checks stably. Renaming a job silently makes the old required check unsatisfiable and blocks every PR until someone notices.

Keeping CI fast

In order of impact:

1. Cancel superseded runs (concurrency). Often halves total minutes.
2. Fail fast on cheap jobs. Lint before matrix.
3. Cache dependencies. Usually the single largest per-job cost.
4. Shrink the PR matrix, move the rest to nightly.
5. Parallelize the test suite (pytest -n auto, cargo nextest, sharded jest).
6. Only run what changed — paths-filter for monorepos, or a task runner with a dependency graph (Nx, Turborepo, Bazel).
7. Skip docs-only changes:

   on:
     pull_request:
       paths-ignore: ['**.md', 'docs/**']
   

   Careful: if the workflow is a required check, paths-ignore makes it never run and never satisfy the requirement. Use a "skip job that reports success" pattern instead.

Public repos get free standard-runner minutes on GitHub-hosted runners; private repos do not. Either way, a 40-minute pipeline costs contributor attention, which is the scarcer resource.

Making CI failures debuggable by strangers

A contributor who cannot understand the failure will abandon the PR.

• Name jobs for what they check: test (ubuntu, py3.13), not build-2.
• Print the reproduction command in the failure path: Run \make test-unit` to reproduce locally.`
• Upload artifacts on failure — logs, screenshots, core dumps:

  - uses: actions/upload-artifact@v4
    if: failure()
    with: { name: logs-${{ matrix.os }}, path: ./logs }
  

• Use annotations so errors appear inline on the diff (most linters have a GitHub reporter; reviewdog wraps the rest).
• Document the flaky jobs you know about and how to tell them apart from real failures. Better: fix them.

Anti-patterns

• No concurrency block. Wasted minutes and contributor waiting time.
• pull_request_target + checkout of PR head. Full compromise; treat as critical.
• Secrets in a job that fork PRs run. They won't work, and the red X is on the contributor.
• Required checks that forks cannot satisfy.
• Unpinned third-party actions. @v1 is a moving target someone else controls.
• A 30-job matrix on every PR.
• Caching build output keyed on the branch name.
• Auto-merge on green with no review. CI checks what you thought to check.
• Self-hosted runners on public repos, non-ephemeral.
• Muting a flaky job instead of fixing it. You just disabled the test suite.