# CI Pipelines

> Design continuous integration for an open-source repository. Use when setting up GitHub Actions from scratch, when CI is slow or flaky, when adding a build matrix across OS and language versions, when configuring caching, or when handling PRs from forks that need secrets. Covers workflow structure, concurrency and cancellation, caching strategy, required checks and branch protection, self-hosted runner risks, and keeping CI cheap. Also use for "our CI takes 40 minutes" or "why is CI failing on forks".

- Skill: `the-open-agent/ci-pipelines` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add the-open-agent/ci-pipelines`
- Raw SKILL.md: https://api.skillmd.com/api/skills/the-open-agent/ci-pipelines/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: the-open-agent (https://skillmd.com/u/the-open-agent)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/the-open-agent/ci-pipelines

---


# CI Pipelines

CI in open source has a constraint that internal CI does not: **most PRs come from
forks, by people who cannot debug your pipeline and will not wait for it.** Design for
that.

## Targets

| Metric | Target | Why |
|---|---|---|
| PR feedback time | **< 10 min** | Beyond this, contributors context-switch away |
| Lint/format feedback | **< 2 min** | Fail fast on the cheap stuff |
| Flake rate | **< 1%** | Above this, red CI gets ignored (see `testing-strategy`) |
| Fork PR success | **100% of non-secret jobs** | A contributor must be able to get green |

The last one is the one projects get wrong. If a fork PR always shows a red X because
a deploy job cannot access secrets, every contributor's first experience is failure.

## Workflow structure

Split by purpose and speed, not into one giant workflow:

```
.github/workflows/
├── ci.yml            # lint + test on PR and push to main — the required check
├── release.yml       # tag-triggered publish (see release-engineering)
├── codeql.yml        # scheduled security scanning
├── nightly.yml       # slow: full matrix, fuzzing, benchmarks, downstream tests
└── docs.yml          # deploy docs on merge to main
```

A minimal, correct `ci.yml`:

```yaml
name: CI
on:
  push:
    branches: [main]
  pull_request:

# Cancel superseded runs — the single biggest CI cost saving, one block long
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

permissions:
  contents: read          # least privilege by default; grant per-job as needed

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '22', cache: 'npm' }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  test:
    needs: lint                      # don't burn matrix minutes on unformatted code
    strategy:
      fail-fast: false               # show every failing cell, not just the first
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node: ['20', '22']
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: ${{ matrix.node }}, cache: 'npm' }
      - run: npm ci
      - run: npm test
```

Details that matter and are usually missing:

- **`concurrency` with `cancel-in-progress`** — a contributor pushing five times in ten
  minutes should not queue five full matrices.
- **`fail-fast: false`** — otherwise one Windows failure hides three others.
- **`permissions: contents: read`** at the top level, elevated only in the job that
  needs it. Default-permissive tokens are the main blast radius in a compromised action.
- **`needs: lint`** — cheap gate first.
- **Pin third-party actions to a SHA**, not a tag (see `supply-chain-security`).
  First-party `actions/*` by major tag is a defensible compromise; anything else, pin.

## Matrix sizing

A full cartesian product is the default and it is almost always wasteful. `3 OS × 4
versions × 2 configs = 24 jobs` for a project whose bugs are all on one axis.

Test the **corners exhaustively and the middle sparsely**:

```yaml
strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest]
    python: ['3.10', '3.13']            # oldest supported + newest
    include:
      - { os: macos-latest,   python: '3.13' }
      - { os: windows-latest, python: '3.13' }
      - { os: ubuntu-latest,  python: '3.12' }
```

Move the exhaustive matrix to `nightly.yml`. PRs get fast signal; the full grid still
runs daily and you learn about the rare cell within 24 hours.

## Caching

Cache the dependency store, never `node_modules`/`.venv` themselves — restoring a
partially valid install directory produces confusing failures.

```yaml
- uses: actions/cache@v4
  with:
    path: ~/.cargo/registry
    key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
    restore-keys: ${{ runner.os }}-cargo-
```

Rules:

- **Key on the lockfile hash.** Keying on anything vaguer serves stale caches.
- **Include the OS and language version in the key**, or macOS restores Linux binaries.
- **`restore-keys` for partial hits** — a near-miss cache still saves most of the time.
- **Language-specific setup actions already do this** (`cache: 'npm'`, `cache: 'pip'`).
  Use them before hand-rolling.
- **Never cache anything that affects correctness** — build outputs keyed loosely
  produce "works in CI, broken for users", the worst failure mode available.
- **GitHub evicts caches after 7 days of no use, 10 GB per repo.** Do not architect
  around a cache being present.

## Fork PRs and secrets

The core security constraint: `pull_request` from a fork gets a **read-only token and
no secrets**. This is correct and you should not fight it.

- **`pull_request_target` runs with full secrets against the base repo** and is the
  most dangerous trigger in GitHub Actions. If you use it, **never check out the PR
  head** — that executes attacker-controlled code with your secrets. Use it only for
  labeling, commenting, and other metadata work.
- **Structure jobs so fork PRs go fully green** without secrets. Guard secret-requiring
  jobs:
  ```yaml
  if: github.event.pull_request.head.repo.full_name == github.repository
  ```
- **Coverage upload, preview deploys, and publishing** all belong in a separate
  workflow triggered by `workflow_run` after CI succeeds, or on push to `main`.
- **Require approval for first-time contributors' workflow runs** — repo Settings →
  Actions. This is the default for new contributors and worth keeping.
- **Self-hosted runners on a public repo are a remote code execution service** unless
  they are ephemeral and isolated. Every fork PR can run arbitrary code on them. If you
  need self-hosted, use ephemeral single-use runners in a disposable VM.

## Required checks and branch protection

Configure on `main`:

- **Require status checks to pass**, and require branches be up to date only if your
  merge volume is low — on a busy repo that setting causes an update-rebase treadmill.
  Prefer a merge queue.
- **Require one approving review**, dismiss stale approvals on new commits.
- **Require conversation resolution.**
- **Include administrators.** If you exempt yourself, the rules are advisory.
- **Do not require jobs that cannot run on forks** — this is the most common way a
  repo becomes unmergeable for outside contributors.

Name required checks stably. Renaming a job silently makes the old required check
unsatisfiable and blocks every PR until someone notices.

## Keeping CI fast

In order of impact:

1. **Cancel superseded runs** (`concurrency`). Often halves total minutes.
2. **Fail fast on cheap jobs.** Lint before matrix.
3. **Cache dependencies.** Usually the single largest per-job cost.
4. **Shrink the PR matrix**, move the rest to nightly.
5. **Parallelize the test suite** (`pytest -n auto`, `cargo nextest`, sharded jest).
6. **Only run what changed** — `paths-filter` for monorepos, or a task runner with a
   dependency graph (Nx, Turborepo, Bazel).
7. **Skip docs-only changes**:
   ```yaml
   on:
     pull_request:
       paths-ignore: ['**.md', 'docs/**']
   ```
   Careful: if the workflow is a required check, `paths-ignore` makes it never run and
   never satisfy the requirement. Use a "skip job that reports success" pattern instead.

Public repos get free standard-runner minutes on GitHub-hosted runners; private repos
do not. Either way, a 40-minute pipeline costs contributor attention, which is the
scarcer resource.

## Making CI failures debuggable by strangers

A contributor who cannot understand the failure will abandon the PR.

- **Name jobs for what they check**: `test (ubuntu, py3.13)`, not `build-2`.
- **Print the reproduction command** in the failure path: `Run \`make test-unit\` to
  reproduce locally.`
- **Upload artifacts on failure** — logs, screenshots, core dumps:
  ```yaml
  - uses: actions/upload-artifact@v4
    if: failure()
    with: { name: logs-${{ matrix.os }}, path: ./logs }
  ```
- **Use annotations** so errors appear inline on the diff (most linters have a GitHub
  reporter; `reviewdog` wraps the rest).
- **Document the flaky jobs you know about** and how to tell them apart from real
  failures. Better: fix them.

## Anti-patterns

- **No `concurrency` block.** Wasted minutes and contributor waiting time.
- **`pull_request_target` + checkout of PR head.** Full compromise; treat as critical.
- **Secrets in a job that fork PRs run.** They won't work, and the red X is on the
  contributor.
- **Required checks that forks cannot satisfy.**
- **Unpinned third-party actions.** `@v1` is a moving target someone else controls.
- **A 30-job matrix on every PR.**
- **Caching build output keyed on the branch name.**
- **Auto-merge on green with no review.** CI checks what you thought to check.
- **Self-hosted runners on public repos**, non-ephemeral.
- **Muting a flaky job instead of fixing it.** You just disabled the test suite.