Cicd Gitlab Pipelines Dev

GitLab CI/CD Pipeline Development

Guide for developing, debugging, and optimizing GitLab CI/CD pipelines.

When to Use This Skill

• Creating new GitLab CI/CD pipelines
• Debugging job failures from pipeline logs
• Understanding GitLab-specific pipeline features
• Optimizing pipeline performance and costs
• Configuring runners and environments
• Working with includes, extends, and templates

Pipeline Structure

File Location

Pipelines are defined in .gitlab-ci.yml at the repository root.

Basic Structure

stages:
  - build
  - test
  - deploy

variables:
  NODE_VERSION: "20"

build:
  stage: build
  image: node:${NODE_VERSION}
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

test:
  stage: test
  image: node:${NODE_VERSION}
  script:
    - npm ci
    - npm test

deploy:
  stage: deploy
  script:
    - ./deploy.sh
  only:
    - main

Key Concepts

Stages

Stages define the order of job execution. Jobs in the same stage run in parallel.

stages:
  - build
  - test
  - deploy
  - cleanup

Jobs

Jobs are the basic unit of execution:

job-name:
  stage: test
  image: ruby:3.2
  before_script:
    - bundle install
  script:
    - bundle exec rspec
  after_script:
    - echo "Cleanup"
  artifacts:
    paths:
      - coverage/
    expire_in: 1 week

Variables

# Global variables
variables:
  DATABASE_URL: "postgres://localhost/test"
  RAILS_ENV: test

# Job-level variables
test:
  variables:
    COVERAGE: "true"
  script:
    - echo $COVERAGE

Predefined Variables

script:
  - echo "Project: $CI_PROJECT_NAME"
  - echo "Branch: $CI_COMMIT_REF_NAME"
  - echo "Commit: $CI_COMMIT_SHA"
  - echo "Pipeline: $CI_PIPELINE_ID"
  - echo "Job: $CI_JOB_ID"
  - echo "Runner: $CI_RUNNER_ID"
  - echo "Environment: $CI_ENVIRONMENT_NAME"

Common CI Patterns

Matrix Builds (Parallel)

test:
  stage: test
  parallel:
    matrix:
      - RUBY_VERSION: ["3.1", "3.2", "3.3"]
        DATABASE: ["postgres", "mysql"]
  image: ruby:$RUBY_VERSION
  script:
    - echo "Testing Ruby $RUBY_VERSION with $DATABASE"

Rules (Conditional Execution)

deploy:
  stage: deploy
  script:
    - ./deploy.sh
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: on_success
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual
    - when: never

DAG (Directed Acyclic Graph)

Use needs for faster pipelines by skipping stage ordering:

stages:
  - build
  - test
  - deploy

build-frontend:
  stage: build
  script: npm run build:frontend

build-backend:
  stage: build
  script: npm run build:backend

test-frontend:
  stage: test
  needs: [build-frontend]  # Runs as soon as build-frontend completes
  script: npm run test:frontend

test-backend:
  stage: test
  needs: [build-backend]
  script: npm run test:backend

deploy:
  stage: deploy
  needs: [test-frontend, test-backend]
  script: ./deploy.sh

Caching

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/

# Or per-job cache
test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  script:
    - npm ci --cache .npm
    - npm test

Artifacts

build:
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    exclude:
      - dist/**/*.map
    expire_in: 1 day
    reports:
      junit: test-results.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura.xml

Includes and Templates

Include External Files

include:
  # Local file
  - local: '/templates/.gitlab-ci-template.yml'

  # From another project
  - project: 'my-group/ci-templates'
    ref: main
    file: '/templates/nodejs.yml'

  # Remote URL
  - remote: 'https://example.com/templates/ci.yml'

  # GitLab templates
  - template: Auto-DevOps.gitlab-ci.yml

Extends (Inheritance)

.test-template:
  image: node:20
  before_script:
    - npm ci
  cache:
    paths:
      - node_modules/

unit-test:
  extends: .test-template
  script:
    - npm run test:unit

integration-test:
  extends: .test-template
  script:
    - npm run test:integration

Reference (YAML Anchors)

.default-rules: &default-rules
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

test:
  <<: *default-rules
  script: npm test

lint:
  <<: *default-rules
  script: npm run lint

Debugging CI Failures

View Pipeline Logs

1. Navigate to CI/CD > Pipelines
2. Click on the pipeline
3. Click on the failed job
4. Review job log output

Debug Mode

test:
  script:
    - npm test
  variables:
    CI_DEBUG_TRACE: "true"  # Verbose logging

Common Issues

Lint Pipeline Locally

# Using GitLab CI Lint API
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data-urlencode "content=$(cat .gitlab-ci.yml)" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/ci/lint"

# Using gitlab-ci-local
npm install -g gitlab-ci-local
gitlab-ci-local --list
gitlab-ci-local test

Runner Configuration

Runner Types

Runner Tags

deploy:
  tags:
    - docker
    - linux
    - production
  script:
    - ./deploy.sh

Docker Executor

test:
  image: python:3.11
  services:
    - postgres:15
    - redis:7
  variables:
    POSTGRES_DB: test
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: ""
  script:
    - pytest

Environment and Secrets

Protected Variables

Configure in Settings > CI/CD > Variables:

• Mark as "Protected" for protected branches only
• Mark as "Masked" to hide in logs

deploy:
  script:
    - echo $DEPLOY_TOKEN  # Masked in logs
  only:
    - main  # Protected variable only available here

Environments

deploy-staging:
  stage: deploy
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - ./deploy.sh staging

deploy-production:
  stage: deploy
  environment:
    name: production
    url: https://example.com
  when: manual
  script:
    - ./deploy.sh production

Performance Optimization

Interruptible Jobs

Cancel running jobs when new commits push:

test:
  interruptible: true
  script:
    - npm test

Resource Groups

Prevent concurrent deploys:

deploy:
  resource_group: production
  script:
    - ./deploy.sh

Workflow Rules

Skip pipelines for certain conditions:

workflow:
  rules:
    - if: $CI_COMMIT_MESSAGE =~ /\[skip ci\]/
      when: never
    - if: $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Pipeline Efficiency

# Use needs for DAG
# Use rules to skip unnecessary jobs
# Cache aggressively
# Use slim images
# Parallelize where possible

test:
  parallel: 4  # Split tests across 4 jobs
  script:
    - npm run test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/adrienverge/yamllint
    rev: v1.35.1
    hooks:
      - id: yamllint
        files: \.gitlab-ci\.yml$
        args: [-c, .yamllint.yml]

Debugging Checklist

• Check .gitlab-ci.yml syntax with CI Lint
• Verify runner is available and has correct tags
• Check job rules/conditions are matching
• Review variable values (use debug trace)
• Verify artifacts are being passed correctly
• Check cache is being used effectively
• Review image availability and credentials
• Check for resource_group conflicts

References

• GitLab CI/CD Documentation
• .gitlab-ci.yml Reference
• Predefined Variables
• CI/CD Examples
• Pipeline Architecture
• Runner Documentation
• gitlab-ci-local - Run pipelines locally