Cicd Pipeline Troubleshooting

CI/CD Pipeline Troubleshooting

Debugs and resolves CI/CD pipeline failures with actionable diagnostics, cache optimization strategies, and build optimization techniques for GitHub Actions, GitLab CI, and other CI platforms.

TL;DR Checklist

• Check job status and exit codes before assuming code issues
• Review cache hit/miss patterns and adjust cache keys appropriately
• Validate artifact paths and permissions before downstream jobs
• Compare build times across pipeline runs to identify bottlenecks
• Enable debug logging only when necessary to avoid log noise
• Verify runner availability and resource constraints
• Test pipeline locally with act or gitlab-runner before committing

When to Use

Use this skill when:

• Pipeline job fails with cryptic error messages requiring diagnostic investigation
• Build times have increased significantly and optimization is needed
• Cache is not being utilized effectively, causing redundant downloads
• Artifacts are missing or inaccessible between pipeline stages
• Parallel builds are failing due to resource contention or race conditions
• Need to compare pipeline performance across different branches or commits

When NOT to Use

Avoid this skill for:

• Code logic errors that don't involve CI infrastructure — use coding skills instead
• Pipeline configuration design from scratch — create a new pipeline first
• Security vulnerability remediation — use security-audit skill instead
• Workflow orchestration across multiple systems — use agent-orchestration skills

Core Workflow

1. Identify Failure Point — Determine which job, stage, or step failed. Checkpoint: Note the exact error message, exit code, and timestamp before investigating.

2. Review Pipeline Configuration — Examine the YAML configuration for syntax errors or misconfiguration. Checkpoint: Validate YAML syntax with yamllint or platform-specific linter.

3. Check Environment Variables — Verify all required variables are set and have correct values. Checkpoint: Confirm variable scope (repository vs. organization vs. environment).

4. Analyze Cache and Dependencies — Assess cache hit rate and dependency installation patterns. Checkpoint: Compare cache key patterns with actual file paths and modification times.

5. Evaluate Build Optimization — Measure build times and identify slow steps or redundant work. Checkpoint: Determine if parallelization or caching improvements are feasible.

6. Verify Artifact Resolution — Confirm artifacts are created, stored, and accessible correctly. Checkpoint: Validate artifact retention period and download permissions.

Implementation Patterns

Pattern 1: GitHub Actions Debugging Commands

Debugging GitHub Actions pipelines requires specific commands to inspect logs, environment, and artifacts.

# Check recent workflow runs
gh run list --limit 10

# View detailed logs for a specific run
gh run view <run-number> --log

# Get job-level logs
gh run view <run-number> --job <job-id>

# Download artifacts from a run
gh run download <run-number> --dir ./artifacts

# List artifacts for a run
gh run list-artifacts <run-number>

# Get workflow run timing
gh run view <run-number> --json createdAt,startedAt,completedAt

# Check runner environment
gh run env

# View workflow file for a run
gh run view <run-number> --workflow

# Check job status in JSON format
gh run view <run-number> --json status,conclusion,startedAt,completedAt

BAD vs GOOD: Debugging Verbose Output

# ❌ BAD: Too verbose, fills logs with noise
- name: Debug everything
  run: |
    set -x
    env
    pwd
    ls -la
    # This floods logs and slows execution

# ✅ GOOD: Targeted debugging with cleanup
- name: Debug cache
  if: always()
  run: |
    echo "Cache key: ${{ env.CACHE_KEY }}"
    echo "Cache paths: ${{ env.CACHE_PATHS }}"
    # Only output what's needed

Pattern 2: GitLab CI Debugging Commands

GitLab CI provides specific tools and commands for pipeline diagnosis and optimization.

# List recent pipeline runs
gitlab pipeline list --project-id <project-id> --limit 10

# Get pipeline details with jobs
gitlab pipeline get <pipeline-id> --project-id <project-id>

# List jobs in a pipeline
gitlab pipeline jobs <pipeline-id> --project-id <project-id>

# Get job logs
gitlab job trace <job-id> --project-id <project-id>

# Download job artifacts
gitlab job artifacts <job-id> --project-id <project-id> --path ./artifacts

# Check pipeline timing
gitlab pipeline get <pipeline-id> --project-id <project-id> --json

# View pipeline graph
gitlab pipeline graph <pipeline-id> --project-id <project-id>

BAD vs GOOD: GitLab CI Debugging

# ❌ BAD: Using echo for everything, hard to parse
debug_job:
  script:
    - echo "Starting debug"
    - echo "ENV_VAR=$ENV_VAR"
    - echo "FILES:"
    - ls -la
    # This creates unstructured log output

# ✅ GOOD: Using structured debug output
debug_job:
  script:
    - echo "::group::Debug Environment"
    - env | sort
    - echo "::endgroup::"
    - echo "::group::Directory Listing"
    - ls -la
    - echo "::endgroup::"
    # Creates collapsible sections in GitLab UI

Pattern 3: Cache Optimization Commands

Proper cache management can reduce build times by 50% or more.

# Check cache hit rate in GitHub Actions
gh api repos/{owner}/{repo}/actions/caches --jq '.actions_caches[] | {key, size_in_bytes, created_at}'

# Delete specific cache by key
gh api repos/{owner}/{repo}/actions/caches --method DELETE \
  -f key=<cache-key>

# Clear all caches for a repository (use carefully!)
gh api repos/{owner}/{repo}/actions/caches --method DELETE \
  --jq '.actions_caches[].id' | while read id; do
    gh api repos/{owner}/{repo}/actions/caches/$id --method DELETE
  done

# For local cache testing (act)
act -n --cache-server-dir ./cache-test

# Check cache size locally
du -sh ~/.cache/act/*

# View act cache contents
act cache list

BAD vs GOOD: Cache Key Patterns

# ❌ BAD: Static cache key never updates
- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-cache  # Always uses same key, never updates

# ✅ GOOD: Dynamic cache key with hash
- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-cache-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      npm-cache-

# ❌ BAD: Too specific cache key misses hits
- uses: actions/cache@v4
  with:
    path: node_modules
    key: npm-${{ github.sha }}-${{ runner.os }}  # Unique per commit

# ✅ GOOD: Balanced cache key strategy
- uses: actions/cache@v4
  with:
    path: |
      node_modules
      .pnpm-store
    key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      npm-${{ runner.os }}-

Pattern 4: Build Time Optimization Commands

Identifying and optimizing slow build steps is critical for CI efficiency.

# Measure GitHub Actions job timing
gh run view <run-number> --json jobs --jq '.jobs[] | {name, startedAt, completedAt}' | \
  jq -r '.[] | "\(.name): \(.completedAt - .startedAt)"'

# Get pipeline duration breakdown
gh run view <run-number> --json steps --jq '
  .steps | group_by(.name) | .[] | 
  {name: .[0].name, total: (map(.completedAt - .startedAt) | add)}
'

# For GitLab CI, check job durations
gitlab job get <job-id> --project-id <project-id> --json | \
  jq '{name, duration, started_at, finished_at}'

# Calculate total pipeline time
gitlab pipeline get <pipeline-id> --project-id <project-id> --json | \
  jq '.duration, .created_at, .finished_at'

# Analyze npm install times
npm install --timing

# View npm timing report
cat ~/.npm/_logs/*.log

BAD vs GOOD: Parallel Build Strategy

# ❌ BAD: Sequential builds that could be parallel
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: npm install
      - run: npm run lint
      - run: npm run test
      - run: npm run build
      # Each step waits for previous

# ✅ GOOD: Parallel build jobs
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - run: npm install
      - run: npm run lint
  test:
    runs-on: ubuntu-latest
    needs: lint
    steps:
      - run: npm install
      - run: npm run test
  build:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - run: npm install
      - run: npm run build
  # Lint and test can run in parallel

Pattern 5: Artifact Resolution Debugging

Artifacts failing to upload or download is a common pipeline failure mode.

# List artifacts for a GitHub Actions run
gh run list-artifacts <run-number>

# Download specific artifact
gh run download <run-number> -n <artifact-name> -d ./download

# Check artifact size limits (GitHub: 10GB per artifact)
gh run view <run-number> --json artifacts --jq '.artifacts[] | {name, size_in_bytes}'

# Check GitLab artifact sizes
gitlab job get <job-id> --project-id <project-id> --json | \
  jq '.artifacts'

# Download large artifacts in chunks
gitlab job artifacts <job-id> --project-id <project-id> \
  --path <specific-path> --output <output-file>

# ❌ BAD: Missing artifact retention and paths
- name: Upload artifacts
  uses: actions/upload-artifact@v4
  with:
    name: my-artifact
    # No path specified, may upload nothing
    retention-days: 1  # Too short for multi-day workflows

# ✅ GOOD: Explicit artifact configuration
- name: Upload build artifacts
  uses: actions/upload-artifact@v4
  with:
    name: build-artifacts-${{ github.sha }}
    path: |
      dist/
      build/
      !dist/**/*.spec.js  # Exclude test files
    retention-days: 7
    if-no-files-found: warn

Pattern 6: Environment and Dependency Debugging

Environment mismatches and dependency issues cause many pipeline failures.

# Check Node.js version in GitHub Actions
gh run view <run-number> --json steps --jq '
  .steps[] | select(.name == "Setup Node") | 
  {name, status, conclusion, outputs}
'

# Verify npm registry configuration
npm config list

# Check yarn registry
yarn config get registry

# Debug Python environment
python -c "import sys; print(sys.executable); print(sys.version)"
pip list | sort
conda list 2>/dev/null || true

# Check Ruby environment
ruby -v
gem env home
bundle exec which ruby

# Debug Docker build cache
docker buildx build --no-cache --progress=plain .

# View build cache details
docker buildx du --verbose

# Prune old build cache
docker buildx prune

BAD vs GOOD: Environment Setup

# ❌ BAD: Not specifying runtime version
- name: Setup Node
  uses: actions/setup-node@v4
  with:
    node-version: 18  # No patch version, may vary

# ✅ GOOD: Specific version and cache configuration
- name: Setup Node
  uses: actions/setup-node@v4
  with:
    node-version: '18.20.3'  # Exact patch version
    cache: 'npm'
    cache-dependency-path: '**/package-lock.json'

Pattern 7: Network and Timeout Debugging

Network failures and timeouts are often misdiagnosed as code issues.

# Check network connectivity in GitHub Actions
- name: Test network
  run: |
    echo "Testing GitHub API connectivity..."
    curl -s -o /dev/null -w "%{http_code}" https://api.github.com
    echo ""
    echo "Testing npm registry..."
    curl -s -o /dev/null -w "%{http_code}" https://registry.npmjs.org

# Increase curl timeout for slow networks
curl --max-time 60 -f https://example.com || echo "Timed out"

# Test Docker registry connectivity
docker login registry.example.com -u user -p token

# ❌ BAD: Hardcoded timeouts without retry
- name: Download dependencies
  run: npm ci
  timeout-minutes: 1  # Too short for large dependencies

# ✅ GOOD: Retry logic with exponential backoff
- name: Download dependencies with retry
  uses: nick-fields/retry@v2
  with:
    timeout_minutes: 5
    max_attempts: 3
    command: npm ci

Pattern 8: Parallel Build Optimization

Optimizing parallel builds requires understanding resource constraints.

# Check parallel job limits in GitHub Actions
gh run view <run-number> --json concurrency --jq '.concurrency'

# List running jobs on a runner
gh api repos/{owner}/{repo}/actions/runs --jq '.workflow_runs[] | select(.status == "in_progress")'

# Check runner status
gh api repos/{owner}/{repo}/actions/runners --jq '.runners[] | {name, status, busy}'

# For self-hosted runners, check CPU and memory
nproc
free -h
df -h

# Monitor resource usage during build
while true; do
  echo "CPU: $(top -bn1 | grep 'Cpu(s)' | awk '{print $2}')%"
  echo "Memory: $(free -m | grep Mem | awk '{print $3 "/" $2 "MB"}')"
  sleep 5
done

BAD vs GOOD: Parallel Strategy

# ❌ BAD: All tests in single job, slow execution
test:
  runs-on: ubuntu-latest
  steps:
    - run: npm install
    - run: npm test
    # All tests run sequentially in one job

# ✅ GOOD: Test matrix for parallel execution
test:
  runs-on: ubuntu-latest
  strategy:
    matrix:
      node-version: [16, 18, 20]
      os: [ubuntu-latest, windows-latest]
  steps:
    - uses: actions/setup-node@v4
      with:
        node-version: ${{ matrix.node-version }}
    - run: npm install
    - run: npm test

Constraints

MUST DO

• Always check job exit codes and error messages before assuming code issues
• Use cache keys that include dependency hashes to ensure cache invalidation
• Implement retry logic for flaky network operations
• Set appropriate artifact retention periods (minimum 7 days for debugging)
• Use parallel builds to reduce total pipeline execution time
• Log cache operations (hits/misses) for optimization analysis
• Validate YAML syntax before committing pipeline changes

MUST NOT DO

• Store secrets in cache keys or expose them in logs
• Use static cache keys that never update (causes stale dependencies)
• Set artifact retention below 3 days (makes debugging difficult)
• Run all tests sequentially in a single job
• Disable job timeouts (can hang pipelines indefinitely)
• Use if: always() without cleanup logic (leaks resources)
• Hardcode platform-specific paths without fallbacks

TL;DR for Code Generation

• Use guard clauses — return early on network failures before proceeding
• Return structured error objects with code, message, and context
• Cache configuration should include version, OS, and dependency hashes
• Implement retry logic with exponential backoff for network operations
• Artifact paths must be absolute or relative to workspace root
• Test parallelization should use matrix strategy for OS and version combinations
• Never store secrets in cache keys or log them inadvertently

Output Template

When debugging a CI/CD pipeline failure, produce:

1. Failure Diagnosis — Exact job, step, and error message with exit code
2. Pipeline Configuration Review — YAML validation results and potential issues
3. Environment Analysis — Runtime versions, dependency versions, and environment variables
4. Cache Analysis — Hit/miss rates, cache key patterns, and optimization recommendations
5. Build Time Analysis — Step-by-step timing with optimization suggestions
6. Artifact Resolution — Upload/download issues and path validation
7. Recommended Fix — Specific commands or configuration changes to resolve

Related Skills

References

• GitHub Actions Documentation: https://docs.github.com/en/actions
• GitLab CI/CD Documentation: https://docs.gitlab.com/ee/ci/
• GitHub CLI Documentation: https://cli.github.com/manual/
• GitLab CLI Documentation: https://docs.gitlab.com/ee/api/
• Cache Optimization Best Practices: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
• GitHub Actions Timeout Settings: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

This skill provides actionable debugging commands and optimization patterns for CI/CD pipelines across multiple platforms.