Validating CI Pipelines Locally

Validating CI Pipelines Locally

This skill is the single source of truth for reproducing a GitLab (or GitHub Actions) pipeline locally. The goal is to catch pipeline failures before pushing to the remote — using the same Docker image, the same env vars, and the same service containers as the real runner.

Goal: Run every job from .gitlab-ci.yml (or .github/workflows/*.yml) on the local machine with results that mirror what the remote runner would produce.

When to Use This Skill

Step 1 — Detect the pipeline format

The repo may use GitLab, GitHub Actions, or both. Detect first:

# GitLab
test -f .gitlab-ci.yml && echo "gitlab" || true
ls .gitlab-ci/*.yml 2>/dev/null    # included files

# GitHub Actions
ls .github/workflows/*.yml 2>/dev/null

If multiple formats exist, the consumer asks the user which to validate. If --ci=<gitlab|github|both> was passed, honour it.

Step 2 — Ensure the runner toolchain is installed

If the local toolchain cannot be installed, fall back to the image-only path. If even that is impossible (no Docker), the consumer must classify Phase 7 as BLOCKED — runner toolchain unavailable and stop.

Verification:

gitlab-runner --version
act --version
docker info >/dev/null 2>&1

Step 3 — Parse the pipeline & enumerate jobs

For GitLab:

# Resolve all included files into a flat job list (yq is the cleanest path)
yq eval '.. | select(has("script") or has("trigger")) | path | .[-1]' .gitlab-ci.yml

# Or, with awk fallback if yq is missing:
grep -E "^[a-zA-Z0-9_-]+:" .gitlab-ci.yml | grep -v "^stages:\|^variables:\|^default:\|^include:" | sed 's/:$//'

For GitHub Actions:

yq eval '.jobs | keys' .github/workflows/*.yml

Order the jobs by stage (stages: field in GitLab; jobs in GitHub default to graph order via needs:). The consumer executes them in that order.

Check whether a job has ever run

A job restricted to only: merge_requests (GitLab) or on: pull_request (GitHub) never executes on a direct push. In a repository where everybody pushes straight to dev/main, such a job can sit broken for months and nobody notices — its first real execution is somebody's first merge request, which then fails for reasons that have nothing to do with their change.

Before trusting a green pipeline history, confirm the test jobs actually ran. This uses the GitLab CLI glab (authenticate once with glab auth login) and jq:

proj="<url-encoded-path>"   # e.g. group%2Fsubgroup%2Fproject

# List the job names of the 10 most recent pipelines — do api:test / app:test appear at all?
glab api "projects/$proj/pipelines?per_page=10" | jq -r '.[].id' | while read -r pid; do
  echo "pipeline $pid:"
  glab api "projects/$proj/pipelines/$pid/jobs" | jq -r '.[] | "  \(.name) [\(.status)]"'
done

# Any merge request ever? (jq prints 0 ⇒ none exist ⇒ merge_requests-only jobs have never run)
glab api "projects/$proj/merge_requests?state=all&per_page=1" | jq 'length'

If the last command prints 0, the merge_requests-only jobs have never been executed. Treat them as unverified, not as passing. Typical breakage found this way: service containers addressed as 127.0.0.1 instead of their service hostname, and cd x && cmd & backgrounding the whole compound so the subsequent cd .. walks out of the workspace.

Step 4 — Resolve the runner image per job

Each job declares an image. The local execution MUST use the same image so dependency versions match:

# Example .gitlab-ci.yml job
test:
  image: node:22.11.0-alpine
  services:
    - mongo:7.0
    - redis:7-alpine
  variables:
    NODE_ENV: ci
    NSC__MONGOOSE__URI: mongodb://mongo:27017/lt-test
  script:
    - pnpm install --frozen-lockfile
    - pnpm run check

Extract:

• image: (mandatory)
• services: (zero or more)
• variables: (job-level + global variables: block, merged with job-level winning)
• before_script: + script: + after_script: (concatenate)
• cache: (key + paths)
• artifacts: (paths + when)

If a job inherits from default: or extends another job, resolve the merged result before execution.

Step 5 — Execute the job locally

Path A — gitlab-runner exec docker (preferred, full fidelity)

gitlab-runner exec docker <job-name> \
  --docker-pull-policy if-not-present \
  --env CI_PROJECT_DIR=/builds/project \
  --env-file .env.ci  # only if you genuinely need extra env; never commit this file

Caveats:

• gitlab-runner exec docker does not natively support services: in newer runner versions (deprecated in 16+). For modern runners, use Path B.
• Cache and artifacts are best-effort — they do not round-trip through GitLab's storage.
• CI_* predefined variables are partially populated; some integration tests that branch on CI_COMMIT_REF_NAME etc may need explicit --env CI_COMMIT_REF_NAME=local.

Path B — Manual reproduction with Docker Compose (modern runner, full services support)

When gitlab-runner exec cannot mount the services, build a minimal docker-compose.ci.yml per pipeline run:

# docker-compose.ci.yml (generated on the fly, do not commit)
services:
  mongo:
    image: mongo:7.0
    healthcheck: { test: ["CMD", "mongosh", "--eval", "db.runCommand('ping')"], interval: 5s }
  redis:
    image: redis:7-alpine
    healthcheck: { test: ["CMD", "redis-cli", "ping"], interval: 5s }
  job:
    image: node:22.11.0-alpine
    depends_on:
      mongo: { condition: service_healthy }
      redis: { condition: service_healthy }
    working_dir: /builds/project
    volumes:
      - .:/builds/project
    environment:
      NODE_ENV: ci
      NSC__MONGOOSE__URI: mongodb://mongo:27017/lt-test
      REDIS_URL: redis://redis:6379
    command: sh -c "<job script joined with && >"

docker compose -f docker-compose.ci.yml run --rm job

This is the most faithful reproduction and works for any runner version.

Path C — act for GitHub Actions

act -j <job-name> \
  --env-file .env.ci.example \
  --container-architecture linux/amd64    # required on Apple Silicon for x86 actions

Use the medium image (-P ubuntu-latest=catthehacker/ubuntu:act-latest) for parity with GitHub-hosted runners.

Step 6 — Env-var handling (never commit secrets)

CI jobs reference secrets via $CI_TOKEN, $DEPLOY_KEY, etc. For local reproduction:

1. List required secrets — grep -E "\\$[A-Z_]+" .gitlab-ci.yml | sort -u.
2. Source from existing local env — ~/.lenneTech/ci-secrets.env (gitignored, user-maintained) is the convention.
3. Generate placeholders for non-sensitive vars only — CI_COMMIT_REF_NAME, CI_PIPELINE_ID, etc.
4. Skip jobs that require unavailable secrets — classify as SKIPPED — secret unavailable: <NAME> rather than fabricating values.

The consumer NEVER writes a real secret into a tracked file or into the prompt.

Step 7 — Cache & Artifact handling

Local runs do not have GitLab's cache/artifact storage. Substitute as follows:

If the pipeline depends on a stage's artifact, run the prior stage first, copy artifacts out, then run the dependent stage with that directory mounted.

Step 8 — Per-job verdict & retry loop

After each job finishes:

For FAIL outcomes, the consumer (when called from /lt-dev:production-ready) attempts remediation:

Re-run the failed job after each remediation. Stop when GREEN or after the configured --max-iterations limit.

Step 9 — Output & evidence

Save the following to tmp/ci-local/:

• <job-name>.log — full stdout/stderr per job
• <job-name>.exit — exit code per job
• report.md — the canonical report block (below)

The directory is gitignored; treat it as scratch state.

Step 10 — Report block (canonical)

Every consumer ends Phase 7 with this block:

### Local CI Pipeline Report

Format: <gitlab|github|both>
Runner: <gitlab-runner|act|docker-compose>
Image parity: <PASS|FAIL>  (mismatch: <comma-separated jobs>)

| Job | Stage | Image | Result | Duration | Iterations | Notes |
|-----|-------|-------|--------|----------|-------------|-------|
| install      | build | node:22.11.0-alpine | PASS    | 42s | 1 | — |
| lint         | test  | node:22.11.0-alpine | PASS    | 18s | 1 | — |
| typecheck    | test  | node:22.11.0-alpine | PASS    | 33s | 1 | — |
| test         | test  | node:22.11.0-alpine | FAIL    | 2m  | 3 | flaky DB seed — fixed in commit abc123 |
| build        | build | node:22.11.0-alpine | PASS    | 1m  | 1 | — |
| audit        | test  | node:22.11.0-alpine | SKIPPED | —   | — | requires CI_TOKEN |

Global verdict: <PASS|PARTIAL|FAIL>
Failing jobs: <comma-separated or "none">
Skipped jobs: <comma-separated or "none">

Cross-Skill References

• Runnability: running-check-script (called from inside any pnpm run check job)
• DevOps configuration baseline: lt-dev:devops-reviewer agent (architectural review of CI configs)
• Server lifecycle (when a job needs an API up): managing-dev-servers
• Production gate (parent): validating-production-readiness (CI PASS is one of the entry criteria)