# Debug Build

> Diagnose failing Buildkite builds in the golang-crossbuild pipeline. Fetches recent failures, reads logs, and traces root causes in Dockerfiles and apt sources lists. Use when builds on main (or another branch) start failing.

- Skill: `elastic/debug-build` (Agent Skill)
- Install (CLI): `npx get-skill add elastic/debug-build`
- Raw SKILL.md: https://api.skillmd.com/api/skills/elastic/debug-build/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: elastic (https://skillmd.com/u/elastic)
- Updated: 2026-11-16
- Page: https://skillmd.com/skills/elastic/debug-build

---


# Debug build failures

Investigate `main` and branches named `N.N` (for example, `1.26` or `1.25`).
If the workflow provides a branch input, investigate only that branch, after
validating that it is `main` or matches `^[0-9]+\.[0-9]+$`.

## MCP tool naming

This skill calls the Buildkite MCP server.  The server is named `buildkite` in both
Claude Code (local) and the GitHub Agentic Workflow, so tool names are always
`mcp__buildkite__<tool_name>` — e.g. `mcp__buildkite__list_builds`.
No changes are needed when switching between local and GH AW execution.

## Step 1 — Find recent builds and eligible branches

Use `mcp__buildkite__list_builds` (org_slug: `elastic`, pipeline_slug:
`golang-crossbuild`, per_page: 100) to find builds from the last seven days on
eligible branches. Follow pagination as needed. Unless a branch input restricts
the analysis, include only `main` and branch names matching
`^[0-9]+\.[0-9]+$`; exclude all other branches. Inspect enough history on each
eligible branch to determine whether each recent failure was subsequently
recovered, including builds newer than that failure.

For each branch, a failed build is relevant only if there is no newer successful
build on that branch in the available build history. If any later build
succeeded, treat the earlier failure as recovered/intermittent and do not
report it. Apply this check to the whole branch, not just adjacent builds or
only the last failure. Failures within the seven-day window without a later
successful build remain reportable. If none remain, report the branch as
healthy.

Record in-window passing builds and only unrecovered failing builds for the
report table. Exclude every failed build followed by a success on the same
branch. Sort the combined table by build creation date descending (newest
first). Use the numeric build number returned by Buildkite as a plain value in
the table. Do not add Buildkite links or URLs in issue text.

## Step 2 — Get the failure summary

Call `mcp__buildkite__get_build_failure_summary` for the most recent relevant failing build
(org_slug: `elastic`, pipeline_slug: `golang-crossbuild`, log_tail: 100).

This single call returns:
- `build.state` and `job_state_counts` — overall picture
- `jobs` — the terminal problem jobs with bounded log tails
- Any annotations and failed test runs

## Step 3 — Reproduce in Docker before classifying the failure

Before declaring any root cause, try to reproduce the failing behavior in a container that matches the affected image/arch. The goal is to distinguish a confirmed cause from a plausible guess based only on log lines.

For each failing job:
- Identify the exact Dockerfile template and target architecture.
- Re-run the smallest equivalent build or package install in Docker using the same Debian version and arch.
- Record whether the same error appears in the container output.
- If a reproduction does not work, keep the diagnosis tentative and continue investigating instead of asserting a root cause.

Minimal reproduction patterns:

### Pattern A — apt dependency conflict (most common)

Reproduce the same package/install path in Docker before concluding it is an apt sources issue. For Debian 12 cross-arch failures such as `go/armhf` or `go/armel`, the relevant reproduction is the actual Dockerfile target, not a generic package install:
```bash
docker build --progress=plain --no-cache --platform linux/armhf \
  -f go/armhf/Dockerfile.tmpl \
  --build-arg VERSION=1.27.1 \
  .
```

If you need a focused probe for the exact package conflict, use the same multi-arch setup that the image performs:
```bash
docker run --rm --platform linux/amd64 debian:12 bash -lc '
  set -eux
  dpkg --add-architecture armel
  dpkg --add-architecture armhf
  apt-get -qq update
  for arch in amd64 armel armhf; do
    echo "--- $arch ---"
    apt-cache madison libpcre2-8-0${arch:+:$arch} 2>/dev/null | grep security || echo "(none)"
  done
  apt-get install -y --no-install-recommends crossbuild-essential-armhf linux-libc-dev-armhf-cross
'
```

This is the failure mode behind the `Multi-Arch: same` errors seen in Debian 12 armhf/armel images: the security repo has amd64 updates while the secondary ports have not yet published matching versions, so `apt` rejects the foreign-arch install. If the failure is tied to a repo/content mismatch, include the affected `sources-debian*.list` file in the reproduction check and verify whether the package version from the repo actually matches the image state.

### Pattern B — Docker build failure (non-apt)

Replay the same Docker build using the exact target, no cached layers, and the affected arch:
```bash
docker build --progress=plain --no-cache --platform linux/amd64 -f go/main/Dockerfile.tmpl .
```

If the build is generated from templates, reproduce the precise image target from the failing Makefile job instead of guessing from the error snippet.

### Pattern C — Go build / test failure

Re-run the same build or test target inside the image used by the job:
```bash
docker run --rm --platform linux/amd64 <image-tag> bash -lc 'make <target> || go test ./...'
```

### Pattern D — Infrastructure / agent failure

A Docker reproduction is not the primary tool here; still check whether the job is transient and whether rerunning the same containerized step is stable.

Only after a Docker reproduction or equivalent targeted build confirms the issue should the workflow classify it as a root cause.

## Step 3A — Classify the failure

Read the log tails and classify the root cause. Common failure patterns in this repo:

### Pattern A — apt dependency conflict (most common)
Log contains lines like:
```
E: Unable to correct problems, you have held broken packages.
<pkg> : Depends: <dep> (= X) but Y is to be installed
```
**Go to Step 4A.**

Do not assume this means stale apt metadata. Exact-version mismatches can also come from
different package versions selected for native and foreign architectures, stale packages
already installed in the base image, or security repositories publishing updates for
architectures at different times. Identify the image, architecture, source list, installed
version, and candidate version before recommending a fix.

### Pattern B — Docker build failure (non-apt)
Log contains `ERROR: failed to build` or `failed to solve` for a reason other than apt.
Identify the failing `Dockerfile:NN` line and read the relevant Dockerfile template.

### Pattern C — Go build / test failure
Log shows `go build` or `go test` errors. Read the relevant source.

### Pattern D — Infrastructure / agent failure
Log shows agent disconnection, timeout, or cloud provider errors. Usually transient — check
if a retry would fix it before digging deeper.

## Step 4A — Diagnose apt dependency conflicts

### 4A-1. Identify which Makefile target failed

The job name tells you: e.g. `Ubuntu X86_64 - Makefile.debian12` → target is `Makefile.debian12`.

Read the relevant Makefile (e.g. `go/Makefile.debian12`) to see which image directories
are built and in what order. Build stops at the first failure.

### 4A-2. Identify the conflicting package and the Dockerfile
The error message names the package (e.g. `libpcre2-8-0`).
The log also shows the Dockerfile line number (e.g. `Dockerfile:23`).
Read the Dockerfile step immediately before the failure too: it may add a foreign
architecture, upgrade packages, or alter apt sources. Build order matters; the first
failing image can prevent later images from running, while a grouped log may contain
multiple independent image builds.

Map the failing Docker image to its template:
- `go/main`     → `go/main/Dockerfile.tmpl`
- `go/base`     → `go/base/Dockerfile.tmpl`
- `go/base-arm` → `go/base-arm/Dockerfile.tmpl`
- `go/armel`    → `go/armel/Dockerfile.tmpl`
- `go/armhf`    → `go/armhf/Dockerfile.tmpl`
- etc.

Read that template. Note:
- Which `RUN apt-get/apt install` blocks exist
- Whether `apt-get update` is run before each install block (missing update = stale package list)
- Which Debian version is in scope (check `sources-debian${DEBIAN_VERSION}.list`)

### 4A-3. Trace installed versions, candidates, and sources per architecture

For the failing image and Debian version, read its effective `sources-debian${VERSION}.list`
and its `FROM` base image. Debian 12+ images may also need to remove the upstream
`/etc/apt/sources.list.d/debian.sources`; check the template's source replacement rather
than assuming the checked-in list is the only active source. Check any `[arch=...]` filters:
the native and foreign architecture may be receiving different indexes by design.

Classify the exact error:
- `Depends: dep (= X) but Y is to be installed`: an exact-version constraint. Find which
  repository and architecture provide both versions; do not infer that an `apt-get update`
  alone will make them compatible.
- `Depends: dep (>= X) but it is not going to be installed` or `not installable`: inspect
  all unsatisfied dependencies, pinning, architecture indexes, and the resolver's candidate
  versions. This can be a transitive dependency such as `libkrb5-3:armel` needing `libssl3:armel`.

When Docker is available, reproduce using the same platform, sources, and order as the
failed Dockerfile. Add the foreign architecture before refreshing indexes, then inspect
both native and target candidates. Replace the sample package with the conflicting runtime
library from the log:
```bash
docker run --rm --platform linux/amd64 debian:12 bash -c '
  printf "deb http://deb.debian.org/debian bookworm main\n" > /etc/apt/sources.list
  rm -f /etc/apt/sources.list.d/debian.sources
  dpkg --add-architecture armel
  apt-get update
  for pkg in libpcre2-8-0 libssl3; do
    for arch in amd64 armel; do
      echo "--- $pkg:$arch ---"
      apt-cache policy "$pkg:$arch"
    done
  done
'
```

Use the actual image's source entries and package set in a real reproduction; the minimal
source list above is only a diagnostic starting point. `apt-cache policy` shows installed,
candidate, and repository versions. `apt-cache madison <package>:<arch>` can further compare
versions across repositories. In a failing Docker layer, also inspect `dpkg-query -W
-f='${Package}:${Architecture} ${Version}\n' <package>` for the installed version.

As of the investigated Debian 12 issue, `go/base/sources-debian12.list` permits security
indexes for amd64, i386, and armhf, while excluding armel. `go/base-arm/sources-debian12.list`
is scoped to arm64. Check the current files; this policy may change. Do not broadly enable or
disable security repositories based only on a package name.

In this incident, a successful x86 build installed `libpcre2-dev` at `10.42-1` from Bookworm
main and later installed `librpm-dev:armhf` successfully. A later build got past the amd64 RPM
install but failed in the armel image because `libkrb5-3:armel` could not resolve `libssl3:armel`.
Another attempt to pin versions in the shared base regressed `go/main`, where security
`libpcre2-dev` required the security-updated exact version of `libpcre2-8-0`. These are
distinct failures; compare the exact failed image and step rather than treating them as one
repeated error.

For any proposed pin:
- Scope it to the affected cross-architecture image, before `dpkg --add-architecture` and
  before package installation; do not put it in a shared base consumed by native images
  unless those consumers have been verified.
- Confirm the pin causes the intended candidate selection and whether `apt-get upgrade`
  needs `--allow-downgrades`.
- Record the security trade-off: pinning a library to Bookworm main can defer its security
  update in that image. Add a concrete recheck condition (compare versions across relevant
  architectures) and treat the pin as temporary, not a permanent Debian policy.
- If package policy is being centralized for multiple Docker build contexts, account for
  each image's build context; Docker `COPY` cannot read a file outside the build context
  without explicitly staging it or changing the context.

### 4A-4. Missing `apt-get update` before install

Check for a missing or stale apt index, but distinguish it from a repository/version skew.
Look for when sources changed, whether apt lists were removed by a previous layer, and whether
the index update ran after adding a foreign architecture. A separate `RUN apt install` can
legitimately use a fresh index retained from an earlier layer; adding `apt-get update` to
every install is not automatically the fix. Reproduce with a clean build or explicit index
refresh, then compare candidate versions.

## Step 5 — Confirm the cause spans multiple builds

After identifying the root cause, confirm it's consistent by checking whether earlier
relevant builds fail with the same error. Use `mcp__buildkite__list_builds` to list more
builds and spot the first relevant failure. Do not use recovered failures (those followed
by a successful build on the same branch) as evidence of an ongoing issue.

## Step 6 — Summarise and propose a fix

Report:
1. Which builds are failing and since when
2. The exact error (package name, version conflict)
3. Whether it's an apt sources issue, missing `apt-get update`, or something else
4. The specific file(s) to change and what to change
5. Any risk or trade-off (e.g. security-update coverage, architecture filters, or temporary package pins)

Ask the user before implementing changes.

## Output contract

When the investigation is complete, follow the separate issue-reporting instruction
file at `.apm/instructions/build-debug-issue.instructions.md`.

That file defines the required GitHub issue title, body sections, and final format.

## Notes

- Keep the scope to this repository only.
- Do not trigger, cancel, or modify any Buildkite builds.
- If there are no unrecovered failures in the lookback window, report the
  eligible branches as healthy and do not characterize recovered failures as
  active failures.

