# Cicd Pipeline Troubleshooting > Diagnoses and resolves CI/CD pipeline failures with actionable debugging commands for GitHub Actions, GitLab CI, and build optimization patterns. - Skill: `paulpas/cicd-pipeline-troubleshooting` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/cicd-pipeline-troubleshooting` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/cicd-pipeline-troubleshooting/raw - Safety review: PASS (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/cicd-pipeline-troubleshooting --- # CI/CD Pipeline Troubleshooting Debugs and resolves CI/CD pipeline failures with actionable diagnostics, cache optimization strategies, and build optimization techniques for GitHub Actions, GitLab CI, and other CI platforms. ## TL;DR Checklist - [ ] Check job status and exit codes before assuming code issues - [ ] Review cache hit/miss patterns and adjust cache keys appropriately - [ ] Validate artifact paths and permissions before downstream jobs - [ ] Compare build times across pipeline runs to identify bottlenecks - [ ] Enable debug logging only when necessary to avoid log noise - [ ] Verify runner availability and resource constraints - [ ] Test pipeline locally with act or gitlab-runner before committing --- ## When to Use Use this skill when: - Pipeline job fails with cryptic error messages requiring diagnostic investigation - Build times have increased significantly and optimization is needed - Cache is not being utilized effectively, causing redundant downloads - Artifacts are missing or inaccessible between pipeline stages - Parallel builds are failing due to resource contention or race conditions - Need to compare pipeline performance across different branches or commits --- ## When NOT to Use Avoid this skill for: - Code logic errors that don't involve CI infrastructure — use coding skills instead - Pipeline configuration design from scratch — create a new pipeline first - Security vulnerability remediation — use security-audit skill instead - Workflow orchestration across multiple systems — use agent-orchestration skills --- ## Core Workflow 1. **Identify Failure Point** — Determine which job, stage, or step failed. **Checkpoint:** Note the exact error message, exit code, and timestamp before investigating. 2. **Review Pipeline Configuration** — Examine the YAML configuration for syntax errors or misconfiguration. **Checkpoint:** Validate YAML syntax with `yamllint` or platform-specific linter. 3. **Check Environment Variables** — Verify all required variables are set and have correct values. **Checkpoint:** Confirm variable scope (repository vs. organization vs. environment). 4. **Analyze Cache and Dependencies** — Assess cache hit rate and dependency installation patterns. **Checkpoint:** Compare cache key patterns with actual file paths and modification times. 5. **Evaluate Build Optimization** — Measure build times and identify slow steps or redundant work. **Checkpoint:** Determine if parallelization or caching improvements are feasible. 6. **Verify Artifact Resolution** — Confirm artifacts are created, stored, and accessible correctly. **Checkpoint:** Validate artifact retention period and download permissions. --- ## Implementation Patterns ### Pattern 1: GitHub Actions Debugging Commands Debugging GitHub Actions pipelines requires specific commands to inspect logs, environment, and artifacts. ```bash # Check recent workflow runs gh run list --limit 10 # View detailed logs for a specific run gh run view --log # Get job-level logs gh run view --job # Download artifacts from a run gh run download --dir ./artifacts # List artifacts for a run gh run list-artifacts # Get workflow run timing gh run view --json createdAt,startedAt,completedAt ``` ```bash # Check runner environment gh run env # View workflow file for a run gh run view --workflow # Check job status in JSON format gh run view --json status,conclusion,startedAt,completedAt ``` #### BAD vs GOOD: Debugging Verbose Output ```yaml # ❌ BAD: Too verbose, fills logs with noise - name: Debug everything run: | set -x env pwd ls -la # This floods logs and slows execution ``` ```yaml # ✅ GOOD: Targeted debugging with cleanup - name: Debug cache if: always() run: | echo "Cache key: ${{ env.CACHE_KEY }}" echo "Cache paths: ${{ env.CACHE_PATHS }}" # Only output what's needed ``` ### Pattern 2: GitLab CI Debugging Commands GitLab CI provides specific tools and commands for pipeline diagnosis and optimization. ```bash # List recent pipeline runs gitlab pipeline list --project-id --limit 10 # Get pipeline details with jobs gitlab pipeline get --project-id # List jobs in a pipeline gitlab pipeline jobs --project-id # Get job logs gitlab job trace --project-id # Download job artifacts gitlab job artifacts --project-id --path ./artifacts ``` ```bash # Check pipeline timing gitlab pipeline get --project-id --json # View pipeline graph gitlab pipeline graph --project-id ``` #### BAD vs GOOD: GitLab CI Debugging ```yaml # ❌ BAD: Using echo for everything, hard to parse debug_job: script: - echo "Starting debug" - echo "ENV_VAR=$ENV_VAR" - echo "FILES:" - ls -la # This creates unstructured log output ``` ```yaml # ✅ GOOD: Using structured debug output debug_job: script: - echo "::group::Debug Environment" - env | sort - echo "::endgroup::" - echo "::group::Directory Listing" - ls -la - echo "::endgroup::" # Creates collapsible sections in GitLab UI ``` ### Pattern 3: Cache Optimization Commands Proper cache management can reduce build times by 50% or more. ```bash # Check cache hit rate in GitHub Actions gh api repos/{owner}/{repo}/actions/caches --jq '.actions_caches[] | {key, size_in_bytes, created_at}' # Delete specific cache by key gh api repos/{owner}/{repo}/actions/caches --method DELETE \ -f key= # Clear all caches for a repository (use carefully!) gh api repos/{owner}/{repo}/actions/caches --method DELETE \ --jq '.actions_caches[].id' | while read id; do gh api repos/{owner}/{repo}/actions/caches/$id --method DELETE done ``` ```bash # For local cache testing (act) act -n --cache-server-dir ./cache-test # Check cache size locally du -sh ~/.cache/act/* # View act cache contents act cache list ``` #### BAD vs GOOD: Cache Key Patterns ```yaml # ❌ BAD: Static cache key never updates - uses: actions/cache@v4 with: path: ~/.npm key: npm-cache # Always uses same key, never updates # ✅ GOOD: Dynamic cache key with hash - uses: actions/cache@v4 with: path: ~/.npm key: npm-cache-${{ hashFiles('**/package-lock.json') }} restore-keys: | npm-cache- ``` ```yaml # ❌ BAD: Too specific cache key misses hits - uses: actions/cache@v4 with: path: node_modules key: npm-${{ github.sha }}-${{ runner.os }} # Unique per commit # ✅ GOOD: Balanced cache key strategy - uses: actions/cache@v4 with: path: | node_modules .pnpm-store key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }} restore-keys: | npm-${{ runner.os }}- ``` ### Pattern 4: Build Time Optimization Commands Identifying and optimizing slow build steps is critical for CI efficiency. ```bash # Measure GitHub Actions job timing gh run view --json jobs --jq '.jobs[] | {name, startedAt, completedAt}' | \ jq -r '.[] | "\(.name): \(.completedAt - .startedAt)"' # Get pipeline duration breakdown gh run view --json steps --jq ' .steps | group_by(.name) | .[] | {name: .[0].name, total: (map(.completedAt - .startedAt) | add)} ' ``` ```bash # For GitLab CI, check job durations gitlab job get --project-id --json | \ jq '{name, duration, started_at, finished_at}' # Calculate total pipeline time gitlab pipeline get --project-id --json | \ jq '.duration, .created_at, .finished_at' ``` ```bash # Analyze npm install times npm install --timing # View npm timing report cat ~/.npm/_logs/*.log ``` #### BAD vs GOOD: Parallel Build Strategy ```yaml # ❌ BAD: Sequential builds that could be parallel jobs: build: runs-on: ubuntu-latest steps: - run: npm install - run: npm run lint - run: npm run test - run: npm run build # Each step waits for previous ``` ```yaml # ✅ GOOD: Parallel build jobs jobs: lint: runs-on: ubuntu-latest steps: - run: npm install - run: npm run lint test: runs-on: ubuntu-latest needs: lint steps: - run: npm install - run: npm run test build: runs-on: ubuntu-latest needs: test steps: - run: npm install - run: npm run build # Lint and test can run in parallel ``` ### Pattern 5: Artifact Resolution Debugging Artifacts failing to upload or download is a common pipeline failure mode. ```bash # List artifacts for a GitHub Actions run gh run list-artifacts # Download specific artifact gh run download -n -d ./download # Check artifact size limits (GitHub: 10GB per artifact) gh run view --json artifacts --jq '.artifacts[] | {name, size_in_bytes}' ``` ```bash # Check GitLab artifact sizes gitlab job get --project-id --json | \ jq '.artifacts' # Download large artifacts in chunks gitlab job artifacts --project-id \ --path --output ``` ```yaml # ❌ BAD: Missing artifact retention and paths - name: Upload artifacts uses: actions/upload-artifact@v4 with: name: my-artifact # No path specified, may upload nothing retention-days: 1 # Too short for multi-day workflows ``` ```yaml # ✅ GOOD: Explicit artifact configuration - name: Upload build artifacts uses: actions/upload-artifact@v4 with: name: build-artifacts-${{ github.sha }} path: | dist/ build/ !dist/**/*.spec.js # Exclude test files retention-days: 7 if-no-files-found: warn ``` ### Pattern 6: Environment and Dependency Debugging Environment mismatches and dependency issues cause many pipeline failures. ```bash # Check Node.js version in GitHub Actions gh run view --json steps --jq ' .steps[] | select(.name == "Setup Node") | {name, status, conclusion, outputs} ' # Verify npm registry configuration npm config list # Check yarn registry yarn config get registry ``` ```bash # Debug Python environment python -c "import sys; print(sys.executable); print(sys.version)" pip list | sort conda list 2>/dev/null || true # Check Ruby environment ruby -v gem env home bundle exec which ruby ``` ```bash # Debug Docker build cache docker buildx build --no-cache --progress=plain . # View build cache details docker buildx du --verbose # Prune old build cache docker buildx prune ``` #### BAD vs GOOD: Environment Setup ```yaml # ❌ BAD: Not specifying runtime version - name: Setup Node uses: actions/setup-node@v4 with: node-version: 18 # No patch version, may vary ``` ```yaml # ✅ GOOD: Specific version and cache configuration - name: Setup Node uses: actions/setup-node@v4 with: node-version: '18.20.3' # Exact patch version cache: 'npm' cache-dependency-path: '**/package-lock.json' ``` ### Pattern 7: Network and Timeout Debugging Network failures and timeouts are often misdiagnosed as code issues. ```bash # Check network connectivity in GitHub Actions - name: Test network run: | echo "Testing GitHub API connectivity..." curl -s -o /dev/null -w "%{http_code}" https://api.github.com echo "" echo "Testing npm registry..." curl -s -o /dev/null -w "%{http_code}" https://registry.npmjs.org ``` ```bash # Increase curl timeout for slow networks curl --max-time 60 -f https://example.com || echo "Timed out" # Test Docker registry connectivity docker login registry.example.com -u user -p token ``` ```yaml # ❌ BAD: Hardcoded timeouts without retry - name: Download dependencies run: npm ci timeout-minutes: 1 # Too short for large dependencies ``` ```yaml # ✅ GOOD: Retry logic with exponential backoff - name: Download dependencies with retry uses: nick-fields/retry@v2 with: timeout_minutes: 5 max_attempts: 3 command: npm ci ``` ### Pattern 8: Parallel Build Optimization Optimizing parallel builds requires understanding resource constraints. ```bash # Check parallel job limits in GitHub Actions gh run view --json concurrency --jq '.concurrency' # List running jobs on a runner gh api repos/{owner}/{repo}/actions/runs --jq '.workflow_runs[] | select(.status == "in_progress")' # Check runner status gh api repos/{owner}/{repo}/actions/runners --jq '.runners[] | {name, status, busy}' ``` ```bash # For self-hosted runners, check CPU and memory nproc free -h df -h # Monitor resource usage during build while true; do echo "CPU: $(top -bn1 | grep 'Cpu(s)' | awk '{print $2}')%" echo "Memory: $(free -m | grep Mem | awk '{print $3 "/" $2 "MB"}')" sleep 5 done ``` #### BAD vs GOOD: Parallel Strategy ```yaml # ❌ BAD: All tests in single job, slow execution test: runs-on: ubuntu-latest steps: - run: npm install - run: npm test # All tests run sequentially in one job ``` ```yaml # ✅ GOOD: Test matrix for parallel execution test: runs-on: ubuntu-latest strategy: matrix: node-version: [16, 18, 20] os: [ubuntu-latest, windows-latest] steps: - uses: actions/setup-node@v4 with: node-version: ${{ matrix.node-version }} - run: npm install - run: npm test ``` --- ## Constraints ### MUST DO - Always check job exit codes and error messages before assuming code issues - Use cache keys that include dependency hashes to ensure cache invalidation - Implement retry logic for flaky network operations - Set appropriate artifact retention periods (minimum 7 days for debugging) - Use parallel builds to reduce total pipeline execution time - Log cache operations (hits/misses) for optimization analysis - Validate YAML syntax before committing pipeline changes ### MUST NOT DO - Store secrets in cache keys or expose them in logs - Use static cache keys that never update (causes stale dependencies) - Set artifact retention below 3 days (makes debugging difficult) - Run all tests sequentially in a single job - Disable job timeouts (can hang pipelines indefinitely) - Use `if: always()` without cleanup logic (leaks resources) - Hardcode platform-specific paths without fallbacks --- ## TL;DR for Code Generation - Use guard clauses — return early on network failures before proceeding - Return structured error objects with code, message, and context - Cache configuration should include version, OS, and dependency hashes - Implement retry logic with exponential backoff for network operations - Artifact paths must be absolute or relative to workspace root - Test parallelization should use matrix strategy for OS and version combinations - Never store secrets in cache keys or log them inadvertently --- ## Output Template When debugging a CI/CD pipeline failure, produce: 1. **Failure Diagnosis** — Exact job, step, and error message with exit code 2. **Pipeline Configuration Review** — YAML validation results and potential issues 3. **Environment Analysis** — Runtime versions, dependency versions, and environment variables 4. **Cache Analysis** — Hit/miss rates, cache key patterns, and optimization recommendations 5. **Build Time Analysis** — Step-by-step timing with optimization suggestions 6. **Artifact Resolution** — Upload/download issues and path validation 7. **Recommended Fix** — Specific commands or configuration changes to resolve --- ## Related Skills | Skill | Purpose | |---|---| | `cncf-tekton` | Kubernetes-native CI/CD pipelines for advanced orchestration | | `cncf-argocd` | GitOps continuous delivery for Kubernetes applications | | `cncf-flux` | GitOps operator for Kubernetes configuration management | --- ## References - GitHub Actions Documentation: https://docs.github.com/en/actions - GitLab CI/CD Documentation: https://docs.gitlab.com/ee/ci/ - GitHub CLI Documentation: https://cli.github.com/manual/ - GitLab CLI Documentation: https://docs.gitlab.com/ee/api/ - Cache Optimization Best Practices: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows - GitHub Actions Timeout Settings: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes --- *This skill provides actionable debugging commands and optimization patterns for CI/CD pipelines across multiple platforms.*