# Pipeline Investigation

> Investigates Buildkite pipeline failures to find root causes. Returns structured JSON to the parent for formatting. Triggers when users ask about failing pipelines, build errors, or need help debugging CI/CD issues. Accepts Buildkite build URLs or build numbers and performs deep investigation.

- Skill: `mock-server/pipeline-investigation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add mock-server/pipeline-investigation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mock-server/pipeline-investigation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: mock-server (https://skillmd.com/u/mock-server)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/mock-server/pipeline-investigation

---


# Buildkite Pipeline Failure Investigation

Investigate Buildkite pipeline failures systematically to identify root causes. This skill uses the Buildkite CLI (`bk`) as the primary interface, with REST API fallback.

## Prerequisites — Buildkite CLI Authentication

The `bk` CLI is the preferred way to interact with Buildkite. Before investigating, ensure it is installed and authenticated.

### Step 0: Check and Install the CLI

```bash
which bk || echo "bk CLI not installed"
```

If `bk` is not installed, install it:

```bash
brew tap buildkite/buildkite && brew install buildkite/buildkite/bk
```

### Step 1: Check Authentication

```bash
bk auth status
```

**If authenticated**, you will see the org slug, token UUID, scopes, and user info. Proceed to investigation.

**If not authenticated**, you will see:
```
Error: you are not authenticated. Run bk auth login to authenticate, or run bk use to select a configured organization
```

In this case, **stop and ask the user** to authenticate:

> Please run `bk auth login` in a terminal to authenticate with Buildkite via browser-based OAuth. This is a one-time setup similar to `aws sso login`. Once done, run `bk auth switch mockserver` to select the organization.

**Do NOT attempt to run `bk auth login` yourself** — it requires interactive browser OAuth that cannot be completed in a non-TTY terminal.

### Step 2: Select Organization (if needed)

If `bk auth status` shows `selected_org: ""`, the org hasn't been selected:

```bash
bk auth switch mockserver
```

### Step 3: Get API Token for REST API Calls

When you need the REST API (for endpoints not covered by `bk` CLI):

```bash
TOKEN=$(bk auth token 2>/dev/null)
if [ -z "$TOKEN" ]; then
  echo "ERROR: bk auth token failed — run 'bk auth login' first"
  exit 1
fi
curl -sH "Authorization: Bearer $TOKEN" "https://api.buildkite.com/v2/..."
```

This extracts the OAuth token from the CLI's keychain storage. Always validate the token is non-empty before using it. **Never ask the user to manually create API tokens** — the CLI handles this automatically.

## Investigation Workflow

### Step 1: Parse Input

Extract organization, pipeline, and build number from the Buildkite URL or user input.

**URL Pattern:**
```
https://buildkite.com/{org}/{pipeline}/builds/{build_number}
```

The organization has multiple pipelines sharing the same agent pool:
- `mockserver` — primary Java build and test
- `mockserver-client-node` — Node.js client
- `mockserver-node` — Node.js module
- `mockserver-performance-test` — performance tests

### Step 2: Get Build Overview

**Using `bk` CLI (preferred):**
```bash
bk build view {build_number} -p {pipeline} --json
```

**Using REST API (fallback):**
```bash
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}"
```

**Save the `commit` SHA and `created_at`** — you will need these later to check whether fixes have already been pushed.

### Step 3: Check Agent Availability

**Using `bk` CLI:**
```bash
bk agent list --json
```

This shows all agents across all pipelines, their connection state, and current job (if busy).

**Key fields to check:**
- `connection_state` — should be `connected`
- `job` — if present, agent is busy; check which pipeline/build it's running
- `meta_data` — agent tags including `queue=default`

### Step 4: List Builds Across All Pipelines

To understand queue contention, check builds across all pipelines sharing the agent pool:

```bash
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/builds?state[]=scheduled&state[]=running&per_page=50"
```

### Step 5: Identify Failed Jobs

```bash
bk build view {build_number} -p {pipeline} --json
```

Filter for failed jobs in the JSON output. Look at `jobs[].state == "failed"`.

### Step 6: Retrieve Job Logs

```bash
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}/jobs/{job_id}/log" \
  | jq -r '.content'
```

**For large logs, save to file:**
```bash
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}/jobs/{job_id}/log" \
  | jq -r '.content' > .tmp/buildkite-{build_number}-log.txt
```

### Step 7: Download and Analyze Build Artifacts

```bash
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}/artifacts" \
  | jq '.[] | {filename,path,download_url}'
```

**Download an artifact:**
```bash
TOKEN=$(bk auth token)
curl -sLH "Authorization: Bearer $TOKEN" \
  "{download_url}" -o .tmp/buildkite-{build_number}-artifact.log
```

### Step 8: Check GitHub Actions (Secondary CI)

If the Buildkite build passed but related GitHub Actions failed (CodeQL):

```bash
gh run list --repo mock-server/mockserver-monorepo --limit 10 --json status,conclusion,name,headBranch,createdAt
gh run view {run_id} --repo mock-server/mockserver-monorepo --log-failed
```

### Step 9: Additional Investigation

**Get build changes (commits that triggered the build):**
```bash
git log --oneline {commit}..HEAD
```

**Check recent build history for patterns:**
```bash
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds?per_page=20&state=failed" \
  | jq '.[] | {number,state,message,created_at}'
```

**Cancel a build (e.g., stuck or blocking):**
```bash
bk build cancel {build_number} -p {pipeline} -y
```

**Rebuild a build:**
```bash
bk build rebuild {build_number} -p {pipeline} -y
```

### Step 10: Confirm Flaky-vs-Real

If the failure looks timing/ordering/port/resource-related (e.g. `Timeout`,
`Connection refused`/`BindException`, intermittent assertion on order or
timing), re-run the single failing test (or check recent builds of the same
commit) to confirm intermittency BEFORE classifying it real-vs-flaky:

```bash
# Re-run only the failing build (does NOT change the commit under test)
bk build rebuild {build_number} -p {pipeline} -y

# Or check whether other builds of the SAME commit passed/failed
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds?commit={commit}&per_page=10" \
  | jq '.[] | {number,state,created_at}'
```

If it passes on re-run (or other builds of the same commit passed), classify it
`FLAKY` and set `reproduced: false`. If it fails again deterministically, it is a
real failure — set `reproduced: true`.

### Step 11: Enumerate Competing Hypotheses

Before concluding a root cause, enumerate the competing hypotheses and the
evidence that rules each out (correlation is not causation — a commit landing
just before the failure is not proof it caused it). Record the survivors and the
ruled-out alternatives in `root_cause.alternative_hypotheses`.

### Step 12: Classify Error

Match the identified root cause against common failure patterns:

| Error Pattern | Category | Scope |
|---|---|---|
| `BUILD FAILURE` in Maven | `BUILD_ERROR` | ISOLATED |
| `Tests run:.*Failures:` | `TEST_FAILURE` | ISOLATED |
| `OutOfMemoryError` | `RESOURCE_ERROR` | MAY_BE_SYSTEMIC |
| `Connection refused` | `NETWORK_ERROR` | MAY_BE_SYSTEMIC |
| `docker: Error` | `DOCKER_ERROR` | ISOLATED |
| `Timeout` | `TIMEOUT` | MAY_BE_SYSTEMIC |
| Passes on re-run / intermittent across builds of same commit | `FLAKY` | ISOLATED (intermittent) |
| Agent did not connect | `AGENT_ERROR` | SYSTEMIC |
| Build `skipped` | `AUTO_SKIPPED` | NORMAL (newer commit superseded) |
| Build `scheduled` (stuck) | `QUEUE_STARVATION` | SYSTEMIC |

### Step 13: Check for Already-Pushed Fixes

**CRITICAL:** Before recommending a fix, check whether the root cause has already been addressed:

```bash
git fetch origin master --quiet
git log --oneline {commit}..origin/master
git diff --name-only {commit}..origin/master
```

**Classify:**

| Classification | Meaning | Report Action |
|---|---|---|
| **ALREADY FIXED** | A commit on `origin/master` addresses this | Mark as fixed, cite the commit |
| **OPEN** | No fix found | Report with recommended fix |

## Build Scheduling and Queue Behaviour

Understanding how Buildkite schedules builds is critical for diagnosing "stuck" builds:

- **`skip_queued_branch_builds`** — if enabled in the Buildkite pipeline settings (check via `bk build view -p {pipeline} --json` under `pipeline.skip_queued_branch_builds`), a newer build for the same branch causes older queued builds to be auto-skipped. This is normal, not a failure. The mockserver pipeline currently has this enabled.
- **`cancel_running_branch_builds`** — if disabled (check via `pipeline.cancel_running_branch_builds`), running builds are NOT automatically cancelled when a newer commit is pushed. Older running builds continue to consume agents. The mockserver pipeline currently has this disabled.
- **Multiple pipelines share the same agent pool** (`queue=default`). Scheduled builds from `mockserver-performance-test`, `mockserver-client-node`, and `mockserver-node` compete with the primary `mockserver` pipeline for agents.
- **ASG max_size** limits the total number of agents. The Lambda scaler cannot add instances beyond this limit. Check the current max with:
  ```bash
  DYLD_LIBRARY_PATH=/opt/homebrew/opt/expat/lib aws autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-names "$(cd terraform/buildkite-agents && terraform output -raw auto_scaling_group_name 2>/dev/null || aws autoscaling describe-auto-scaling-groups --profile mockserver-build --region eu-west-2 --query 'AutoScalingGroups[?contains(Tags[?Key==`Name`].Value, `buildkite-mockserver`)].AutoScalingGroupName' --output text | head -1)" \
    --region eu-west-2 --profile mockserver-build \
    --query 'AutoScalingGroups[0].{MinSize:MinSize,MaxSize:MaxSize,DesiredCapacity:DesiredCapacity}'
  ```

## Output — Structured Data Return

Return this structure in your final message:

```json
{
  "schema": "pipeline-investigation/v1",
  "build": {
    "number": 0,
    "pipeline": "<pipeline slug>",
    "branch": "<branch>",
    "commit": "<commit sha>",
    "state": "failed",
    "failed_job": "<job name>",
    "started_at": "<ISO8601>",
    "finished_at": "<ISO8601>"
  },
  "reproduced": true,
  "root_cause": {
    "summary": "<one-line description>",
    "detail": "<technical explanation>",
    "error_excerpt": "<relevant log lines>",
    "failure_category": "<category from table above (e.g. FLAKY) or null>",
    "scope": "SYSTEMIC|ISOLATED|MAY_BE_SYSTEMIC",
    "alternative_hypotheses": [
      { "hypothesis": "<competing explanation>", "ruled_out_by": "<evidence that excludes it>" }
    ]
  },
  "fix_status": "OPEN|ALREADY_FIXED",
  "fix_commit": "<sha or null>",
  "fix_message": "<commit message or null>",
  "github_actions": [
    { "run_id": 0, "workflow": "<name>", "conclusion": "failure|success", "root_cause": "<summary or null>" }
  ],
  "commands_run": ["<exact bk / curl / git command used to gather evidence>"],
  "recommended_fix": "<actionable steps, null if already fixed>"
}
```

After returning the JSON, provide a brief summary (2-3 lines).

## Notes

- Always investigate the deepest failure first (root cause in logs, not surface symptoms)
- Build artifacts often contain more detailed logs than the console output
- For recurring failures, check recent commits and pipeline definition changes
- **Always run Step 13** before reporting — check if the issue was already fixed
- When builds are stuck/scheduled, check agent availability across ALL pipelines, not just the one being investigated
- Set `reproduced: true` only when the failure recurs deterministically (failed again on re-run, or every build of the commit failed); set `reproduced: false` for `FLAKY` failures that passed on re-run
- Always populate `commands_run` with the exact commands used so the conclusion can be replayed by the reader
- The `bk` CLI uses `-p {pipeline}` for pipeline selection, NOT `--org` — the org is set via `bk auth switch`