Circleci

CircleCI

Purpose

Define CI/CD pipelines using CircleCI configuration with workflows, jobs, orbs, caching strategies, and parallel execution for fast, reliable builds.

Architecture Decision Trees

Executor Selection

Caching Strategy

Test Splitting Methods

Workflow Triggers

Quick Start

config.yml with default Docker executor → checkout → restore_cache → install deps → run tests → save_cache → store_artifacts → workflow with lint-test-deploy stages.

Core Workflow

Step 1: Standard Configuration

# .circleci/config.yml
version: 2.1

orbs:
  node: circleci/node@5.1.0
  aws-cli: circleci/aws-cli@3.1.4
  slack: circleci/slack@4.12.5

parameters:
  deploy-prod:
    type: boolean
    default: false

executors:
  node-executor:
    docker:
      - image: cimg/node:22.0
    resource_class: medium
  machine-executor:
    machine:
      image: ubuntu-2204:current
    resource_class: medium

commands:
  setup-dependencies:
    steps:
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Verify dependencies
          command: |
            npm audit --audit-level=high
            npx license-checker --summary
  build-app:
    steps:
      - run: npm run build
      - persist_to_workspace:
          root: .
          paths:
            - dist/
            - node_modules/

jobs:
  lint:
    executor: node-executor
    steps:
      - checkout
      - setup-dependencies
      - run: npm run lint

  test:
    executor: node-executor
    parallelism: 4
    steps:
      - checkout
      - setup-dependencies
      - run:
          name: Run tests with splitting
          command: |
            circleci tests glob "tests/**/*.test.ts" | \
              circleci tests run --command="xargs npx jest --ci --runInBand" \
              --split-by=timings
      - store_test_results:
          path: test-results/
      - store_artifacts:
          path: test-results/
          destination: test-reports

  build:
    executor: machine-executor
    steps:
      - checkout
      - setup-dependencies
      - build-app
      - run:
          name: Build Docker image
          command: |
            docker build -t myapp:$CIRCLE_SHA1 .
            echo "$DOCKER_PASS" | docker login -u "$DOCKER_USER" --password-stdin
            docker push myapp:$CIRCLE_SHA1
      - store_artifacts:
          path: dist/
          destination: build-output

  deploy-staging:
    executor: machine-executor
    steps:
      - attach_workspace:
          at: .
      - aws-cli/setup:
          aws-access-key-id: AWS_ACCESS_KEY_ID
          aws-secret-access-key: AWS_SECRET_ACCESS_KEY
          region: us-east-1
      - run:
          name: Deploy to staging
          command: |
            aws ecs update-service --cluster staging \
              --service myapp --force-new-deployment

  deploy-production:
    executor: machine-executor
    steps:
      - slack/notify:
          event: pending
          message: "Production deploy starting..."
      - deploy-staging
      - slack/notify:
          event: success
          message: "Production deploy completed successfully"
      - slack/notify:
          event: fail
          message: "Production deploy FAILED"

  security-scan:
    executor: node-executor
    steps:
      - checkout
      - node/install-packages
      - run: npm audit --audit-level=high
      - run:
          name: Snyk security scan
          command: |
            curl -sL https://snyk.io/cli/install | bash
            snyk test --severity-threshold=high

  nightly-e2e:
    machine: true
    steps:
      - checkout
      - run: npm ci
      - run: npm run test:e2e
      - store_artifacts:
          path: cypress/videos/

workflows:
  version: 2

  ci-pipeline:
    unless: << pipeline.parameters.deploy-prod >>
    jobs:
      - lint
      - test:
          requires: [lint]
      - build:
          requires: [test]
      - security-scan:
          requires: [lint]
      - deploy-staging:
          requires: [build, security-scan]
          filters:
            branches:
              only: main
      - hold-production:
          type: approval
          requires: [deploy-staging]
      - deploy-production:
          requires: [hold-production]
          filters:
            branches:
              only: main

  nightly-tests:
    triggers:
      - schedule:
          cron: "0 3 * * *"
          filters:
            branches:
              only: main
    jobs:
      - test
      - nightly-e2e

  production-deploy:
    when: << pipeline.parameters.deploy-prod >>
    jobs:
      - deploy-production

Step 2: Dynamic Configuration with Pipeline Parameters

# Trigger with API:
# curl -X POST https://circleci.com/api/v2/project/gh/org/repo/pipeline \
#   -H "Circle-Token: $TOKEN" \
#   -d '{"parameters": {"deploy-prod": true}}'

# In config.yml, use pipeline parameters
parameters:
  deploy-prod:
    type: boolean
    default: false
  service-name:
    type: string
    default: "all"

workflows:
  version: 2
  deploy-service:
    when: << pipeline.parameters.deploy-prod >>
    jobs:
      - deploy:
          name: "deploy-<< pipeline.parameters.service-name >>"

Step 3: Matrix Build for Multi-Version

# Matrix jobs
test-matrix:
  parameters:
    node-version:
      type: string
  executor:
    docker:
      - image: cimg/node:<< parameters.node-version >>
  steps:
    - checkout
    - run: npm ci
    - run: npm test

workflows:
  test-all-versions:
    jobs:
      - test-matrix:
          matrix:
            parameters:
              node-version: ["18.0", "20.0", "22.0"]

Step 4: Custom Orbs

# .circleci/orbs/my-orb/orb.yml
version: 2.1
description: "Custom deployment orb"

commands:
  deploy:
    parameters:
      environment:
        type: enum
        enum: ["dev", "staging", "prod"]
      cluster:
        type: string
    steps:
      - run:
          name: Deploy to << parameters.environment >>
          command: |
            aws ecs update-service \
              --cluster << parameters.cluster >> \
              --service myapp-<< parameters.environment >> \
              --force-new-deployment
      - run:
          name: Wait for deployment
          command: |
            aws ecs wait services-stable \
              --cluster << parameters.cluster >> \
              --services myapp-<< parameters.environment >>

  canary-deploy:
    parameters:
      percent:
        type: integer
        default: 10
    steps:
      - run:
          command: |
            echo "Deploying to << parameters.percent >>% of traffic"
            # Canary logic here

executors:
  aws-executor:
    docker:
      - image: cimg/python:3.11
    environment:
      AWS_REGION: us-east-1

Step 5: Optimization Configuration

# Performance optimization
jobs:
  test:
    docker:
      - image: cimg/node:22.0
    resource_class: medium+  # 4 vCPU, 8GB RAM
    parallelism: 8
    steps:
      - checkout
      # Dependency caching
      - restore_cache:
          keys:
            - v1-deps-{{ checksum "package-lock.json" }}
            - v1-deps-
      - run: npm ci
      - save_cache:
          key: v1-deps-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
            - ~/.npm
      # Test with parallelization
      - run: |
          TESTFILES=$(circleci tests glob "tests/**/*.test.ts" | circleci tests split)
          npx jest --ci --maxWorkers=2 $TESTFILES
      - store_test_results:
          path: test-results/

Anti-Patterns

Anti-Pattern 1: No Resource Class Optimization

Using default medium for all jobs. Build and test jobs benefit from medium+ (faster) or arm.medium (cheaper for ARM builds).

Anti-Pattern 2: Monolithic Config File

Single huge config.yml. Split into reusable orbs, commands, and executors for composability.

Anti-Pattern 3: Ignoring Test Splitting

Running all tests serially on one container. Use parallelism: N with circleci tests split for significant time reduction.

Anti-Pattern 4: No Caching

Reinstalling dependencies every build. Use save_cache/restore_cache with appropriate key patterns.

Anti-Pattern 5: Hardcoded Secrets

Putting environment variables in config.yml. Use CircleCI contexts and project environment variables.

Production Considerations

• Use contexts to share environment variables across projects.
• Use SSH debug (circleci local execute --job test) for troubleshooting.
• Use custom Docker images (Dockerfile + cimg/base) for faster executor startup.
• Set resource class based on job requirements (CPU/memory).
• Monitor credit usage and set budget alerts.
• Configure auto-cancel redundant builds on branch pushes.

Rules & Constraints

• Use version 2.1 configuration format.
• Cache dependencies with checksum-based keys.
• Use contexts for shared secrets across projects.
• Store test results and artifacts for every run.
• Use SSH debug jobs for troubleshooting — never for production.

References

• references/circleci-advanced.md
• references/circleci-fundamentals.md
• references/config-structure.md
• references/kubernetes-integration.md
• references/orb-ecosystem.md
• references/performance-optimization.md
• references/matrix-builds-guide.md

Handoff

Next: cicd-pipeline — general pipeline concepts beyond CircleCI.

Implementation Patterns

YAML: Parallel Test Sharding with Orb

version: 2.1
orbs:
  node: circleci/node@5.1.0

commands:
  restore-workspace:
    steps:
      - attach_workspace:
          at: ~/project

jobs:
  test-sharded:
    executor: node/default
    parallelism: 4
    steps:
      - checkout
      - restore-workspace
      - run: npm ci
      - run:
          command: |
            TEST_FILES=$(circleci tests glob "tests/**/*.test.ts" | circleci tests split --split-by=timings)
            npx jest --ci --runInBand $TEST_FILES
      - store_test_results:
          path: junit.xml

  deploy:
    executor: node/default
    steps:
      - checkout
      - setup_remote_docker
      - run: |
          docker build -t $DOCKER_REGISTRY/app:$CIRCLE_SHA1 .
          docker push $DOCKER_REGISTRY/app:$CIRCLE_SHA1
      - run: |
          ssh deploy@$HOST "kubectl set image deployment/app \
            app=$DOCKER_REGISTRY/app:$CIRCLE_SHA1"

workflows:
  version: 2
  ci:
    jobs:
      - test-sharded
      - deploy:
          requires:
            - test-sharded
          filters:
            branches:
              only: main

Bash: CircleCI API-driven Pipeline Trigger

#!/usr/bin/env bash
trigger_pipeline() {
  local project_slug=$1
  local branch=$2

  curl -X POST \
    "https://circleci.com/api/v2/project/${project_slug}/pipeline" \
    -H "Circle-Token: ${CIRCLE_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "{
      \"branch\": \"${branch}\",
      \"parameters\": {
        \"run_integration_tests\": true,
        \"deploy_to_staging\": true
      }
    }"
}

Production Considerations

• Set resource_class per job based on workload needs; use medium+ for test, large for builds
• Configure pipeline-level environment variables in CircleCI UI, never in config files
• Use contexts to share environment variables across projects with restricted access
• Set up Slack/Webhook notifications for failed pipelines with links to the failing step
• Implement pipeline auto-cancel for redundant builds on non-default branches
• Configure SSH key fingerprints in CircleCI and bind them per-job for production access
• Store Docker credentials in project environment variables, not in the config file

Anti-Patterns

• Checking in API tokens or SSH keys to the repository — always use CircleCI contexts
• Running Docker builds without remote Docker — leads to cache invalidation on every run
• Using machine executor when docker executor suffices — increases setup time
• Ignoring test splitting — running the full suite on one container slows pipelines unnecessarily
• Hardcoding resource_class without monitoring actual usage — overpaying for idle capacity
• Storing large artifacts without retention policies — fills up workspace and storage
• Using setup_remote_docker on every job when one shared Docker context suffices

Performance Optimization

• Enable Docker Layer Caching (DLC) for builds using setup_remote_docker
• Use circleci tests glob with --split-by=timings for optimal test distribution
• Cache node_modules or dependency directories with save_cache/restore_cache by lockfile checksum
• Use workspace persistence for sharing built artifacts between jobs instead of re-checking out
• Configure resource_class auto-scaling based on queue depth for bursting workloads
• Reduce job count by merging quick sequential steps into a single job
• Set no_output_timeout appropriately per step; don't let hung jobs run indefinitely

Security Considerations

• Enable SaaS or self-hosted runner IP allowlisting for your VCS
• Rotate project API tokens when a team member departs
• Use Docker content trust to verify image signatures before pulling base images
• Restrict context access to specific projects and teams in CircleCI organization settings
• Scan base images nightly and trigger rebuilds when vulnerabilities are found
• Never export environment variables in pipeline logs — mask sensitive values
• Require approval jobs for production deployments with manual gate checks