Pipeline Authoring

Pipeline Authoring

Help users write and improve Buildkite pipeline configurations.

When to use

• "Help me write a pipeline"
• "Create a CI pipeline for X"
• "Add a step to my pipeline"
• "How do I do X in Buildkite?"
• "Optimize my pipeline"
• "Add matrix builds"
• "Set up parallel testing"

Available MCP Tools

Also use file reading to examine existing pipeline.yml or .buildkite/ files in the user's codebase.

Approach

1. Understand the goal

   • What workflow are they building?
   • What language/framework?
   • What's the deployment target?

2. Check existing configuration

   • Look for .buildkite/pipeline.yml or pipeline.yml
   • Use buildkite_get_pipeline if they have an existing pipeline

3. Provide appropriate help

   • New pipeline: scaffold complete config
   • Modification: suggest specific changes
   • Question: explain with examples

4. Validate the result

   • Check dependencies make sense
   • Verify conditions are correct
   • Consider edge cases

Pipeline Structure

# Environment for all steps
env:
  NODE_ENV: "test"

# Default agent configuration
agents:
  queue: "default"

# Pipeline steps
steps:
  - command: "npm test"
    label: ":npm: Tests"

Step Types

Command Step

- label: ":test_tube: Tests"
  command: "npm test"
  # Multiple commands
  commands:
    - "npm ci"
    - "npm test"
  # Artifacts to upload
  artifact_paths:
    - "coverage/**/*"
  # Step-specific env
  env:
    CI: "true"
  # Timeout
  timeout_in_minutes: 10
  # Retry configuration
  retry:
    automatic:
      - exit_status: -1
        limit: 2

Wait Step

- wait

# Continue even if previous failed
- wait:
    continue_on_failure: true

Block Step

- block: ":rocket: Deploy"
  prompt: "Ready to deploy?"
  branches: "main"
  fields:
    - text: "Reason"
      key: "reason"
      required: true
    - select: "Environment"
      key: "env"
      options:
        - label: "Staging"
          value: "staging"
        - label: "Production"
          value: "production"

Trigger Step

- trigger: "deploy-pipeline"
  label: ":rocket: Deploy"
  build:
    branch: "${BUILDKITE_BRANCH}"
    commit: "${BUILDKITE_COMMIT}"
    env:
      DEPLOY_ENV: "production"

Group Step

- group: ":test_tube: Tests"
  steps:
    - command: "npm run test:unit"
      label: "Unit"
    - command: "npm run test:e2e"
      label: "E2E"

Dependencies

Explicit with Keys

steps:
  - command: "npm ci"
    key: "install"

  - command: "npm test"
    key: "test"
    depends_on: "install"

  - command: "npm run build"
    depends_on:
      - "install"
      - "test"

Allow Dependency Failure

- command: "npm run report"
  depends_on:
    - step: "test"
      allow_failure: true

Conditionals

Branch Filtering

- command: "deploy.sh"
  branches: "main"

# Multiple branches
- command: "deploy.sh"
  branches:
    - "main"
    - "release/*"

If Conditions

# Only on main
- command: "deploy.sh"
  if: build.branch == "main"

# Only for tags
- command: "release.sh"
  if: build.tag =~ /^v[0-9]+/

# Skip draft PRs
- command: "full-tests.sh"
  if: build.pull_request.draft != true

# Based on metadata
- command: "deploy.sh"
  if: build.meta_data.deploy == "true"

Parallelism and Matrix

Parallel Jobs

- command: "npm test"
  parallelism: 4
  # Uses BUILDKITE_PARALLEL_JOB (0-3)
  # and BUILDKITE_PARALLEL_JOB_COUNT (4)

Matrix Builds

- command: "test.sh"
  matrix:
    setup:
      node: ["16", "18", "20"]
      os: ["linux", "macos"]
  # Creates 6 jobs (all combinations)
  agents:
    os: "{{matrix.os}}"
  env:
    NODE_VERSION: "{{matrix.node}}"

Matrix with Adjustments

- command: "test.sh"
  matrix:
    setup:
      node: ["16", "18", "20"]
    adjustments:
      - with:
          node: "20"
        soft_fail: true

Best Practices

Use Keys for Dependencies

# Good - explicit
- command: "build"
  key: "build"
- command: "test"
  depends_on: "build"

# Avoid - implicit ordering is fragile

Soft Fail for Non-Critical

- command: "npm run lint"
  soft_fail: true

# Or specific exit codes
- command: "npm audit"
  soft_fail:
    - exit_status: 1

Retry Flaky Operations

- command: "npm test"
  retry:
    automatic:
      - exit_status: -1  # Agent lost
        limit: 2
      - exit_status: 255 # Network
        limit: 2

Always Set Timeouts

- command: "npm test"
  timeout_in_minutes: 15

Cache Dependencies

- command: "npm test"
  plugins:
    - cache#v1.0.0:
        paths:
          - "node_modules"
        key: "npm-{{ checksum 'package-lock.json' }}"

Common Patterns

Basic CI

steps:
  - label: ":npm: Install"
    command: "npm ci"
    key: "install"

  - label: ":eslint: Lint"
    command: "npm run lint"
    depends_on: "install"
    soft_fail: true

  - label: ":jest: Test"
    command: "npm test"
    depends_on: "install"

  - label: ":package: Build"
    command: "npm run build"
    depends_on:
      - "install"
      - "test"

Deploy Pipeline

steps:
  - label: ":test_tube: Test"
    command: "npm test"
    key: "test"

  - wait

  - label: ":rocket: Deploy Staging"
    command: "deploy.sh staging"
    key: "staging"
    branches: "main"

  - block: ":rocket: Deploy Production?"
    branches: "main"

  - label: ":rocket: Deploy Production"
    command: "deploy.sh production"
    branches: "main"

Monorepo

steps:
  - label: ":mag: Detect Changes"
    command: "scripts/detect-changes.sh"
    key: "detect"

  - label: ":package: Build API"
    command: "npm run build -w api"
    if: build.env.API_CHANGED == "true"
    depends_on: "detect"

  - label: ":package: Build Web"
    command: "npm run build -w web"
    if: build.env.WEB_CHANGED == "true"
    depends_on: "detect"

Response Format

When helping with pipelines:

1. Understand what they're trying to achieve
2. Show the YAML configuration
3. Explain key parts and why they're configured that way
4. Validate the logic makes sense
5. Suggest improvements if applicable

Always provide complete, copy-pasteable YAML.

Example Interaction

User: Help me set up CI for my Node.js project

1. Ask about test framework, deploy targets, any special needs
2. Provide scaffold:

​```yaml
steps:
  - label: ":npm: Install"
    command: "npm ci"
    key: "install"

  - label: ":jest: Test"
    command: "npm test"
    depends_on: "install"
    artifact_paths:
      - "coverage/**/*"

  - label: ":package: Build"
    command: "npm run build"
    depends_on: "install"
    artifact_paths:
      - "dist/**/*"
​```

3. Explain the structure and ask if they need additions

Converted and distributed by TomeVault — claim your Tome and manage your conversions.