# Pipeline Triage Workflow

> Ordered agent-style triage for a single failed Jenkins build - gather metadata, logs, classification, summary

- Skill: `coreos/pipeline-triage-workflow` (Agent Skill)
- Install (CLI): `npx skillmds@latest add coreos/pipeline-triage-workflow`
- Raw SKILL.md: https://api.skillmd.com/api/skills/coreos/pipeline-triage-workflow/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: coreos (https://skillmd.com/u/coreos)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/coreos/pipeline-triage-workflow

---


# Pipeline Triage Workflow (agentic)

Run **one failed build** through a **fixed sequence** of stages. Each stage has a **single job** and **structured output** for the next. Do **not** stop between stages for "what next?"

**Inputs (required):** `JOB` (Jenkins job name), `BUILD` (integer build number).

**Related skills:** `pipeline-failures` (deep patterns, kola analysis), `pipeline-jira` (COS ticket formatting).

---

## Jenkins Job Hierarchy (know before you start)

| Job | Has downstream? | Analysis approach |
|-----|-----------------|-------------------|
| `build` | **Yes** → triggers `build-arch` per-arch | Check console for which arch failed, then analyze that `build-arch` child job |
| `build-arch` | **No** (leaf job) | Analyze directly - kola tests run here |
| `build-node-image` | **No** (independent pipeline) | Analyze directly - does NOT trigger `build-arch` |

**Critical:** `build-node-image` is a **separate pipeline** from `build`/`build-arch`. If investigating a `build-node-image` failure:
- Analyze its console log directly
- Do NOT search for "downstream" `build-arch` jobs - there are none
- A `build-arch` job running at the same time is **coincidental**, not related

**Stream validation:** Always verify streams match. A failure in `build-node-image` for stream `4.21-9.6` cannot be caused by `build-arch` for stream `rhel-10.2`.

---

## Stage 1 — Gather (build metadata)

**Agent role:** Collect facts about the failing build.

**Run:**
```bash
coreos-tools jenkins builds info <JOB> <BUILD>
coreos-tools jenkins jobs info <JOB>
```

**Output (write this block before Stage 2):**

```markdown
### Gather
- **Job:** …
- **Build:** …
- **Status / result:** …
- **Stream / parameters (if any):** …
- **URL:** …
```

---

## Stage 2 — Logs (console evidence)

**Agent role:** Pull console log and extract high-signal lines.

**Run:**
```bash
coreos-tools jenkins builds log <JOB> <BUILD> | jq -r '.console_log[]' > /tmp/build.log
```

If the log is huge, capture tail:
```bash
coreos-tools jenkins builds log <JOB> <BUILD> | jq -r '.console_log[]' | tail -n 200
```

**Output:**

```markdown
### Logs (excerpt)
- **Last ~N lines or key errors:** …
- **Patterns seen** (error / timeout / registry / test): …
```

Use grep patterns from `pipeline-failures` when analyzing saved log text (`error:`, `FATAL:`, `timeout`, `unauthorized`, `FAILED`, etc.).

**Optional:** If kola tests failed, add kola summary:
```bash
coreos-tools jenkins builds kola-failures <JOB> <BUILD>
```

---

## Stage 3 — Classify

**Agent role:** Map the failure to a **single primary** category and note confidence.

**Categories (pick one primary):** `infrastructure` | `flake` | `test_regression` | `package_change` | `registry_auth` | `tooling` | `unknown`

**Rules of thumb:**
- Transient network / GitLab / "try rerun" language → **flake** or **infrastructure**
- `unauthorized` pulling images → **registry_auth**
- Kola `rerun_failed: true` (if you have kola output) → **test_regression** / **package_change** per `pipeline-failures`
- Only flakes on rerun → do **not** treat as sole root cause; look for compose errors in log

**Output:**

```markdown
### Classify
- **Primary:** …
- **Confidence:** low | medium | high
- **Why (1–3 bullets):** …
```

---

## Stage 4 — Summarize (triage conclusion)

**Agent role:** Produce the **handoff package** for humans (and later Jira).

**Output:**

```markdown
### Triage summary
- **Summary:** One paragraph describing the failure, root cause, and impact.
- **Evidence:** build URL, log pointers
- **Suggested next steps:** (e.g. rerun / open COS subtask / escalate to RHEL / snooze test)
- **Related downstream jobs to check:** e.g. `build-arch` if `build` failed (see `pipeline-failures`)
```

---

## Stage 5 — Upstream Links (optional but recommended)

**Agent role:** Find relevant upstream references for the failure.

**When to search:**
- Test failures → find test source code
- Package changes → find Brew build, changelog
- cosa changes → find coreos-assembler commits
- Known issues → search for existing GitHub/GitLab issues

**Load skills:**
- `rhcos-repositories` — repo locations and test paths
- `rhcos-brew` — package build info
- `pipeline-failures` — has `gh search` commands

**Commands:**

```bash
# Find test source (use rhcos-repositories to determine which repo)
gh search code "<test-name>" --repo coreos/coreos-assembler --repo openshift/os \
  --json repository,path,url

# Search for related issues
gh search issues "<error-pattern>" --repo coreos/coreos-assembler --repo openshift/os \
  --json repository,title,url,state

# Get Brew build info (if package change)
brew buildinfo <package-nvr>
```

**Output:**

```markdown
### Upstream Links
- **Test source:** <url to test file>
- **Related issues:** <issue URLs or "None found">
- **Related PRs:** <PR URLs or "None found">
- **Package build:** <Brew URL if applicable>
```

---

## Execution rules

1. Complete **Stages 1–5 in order** in one run when the user provides `JOB` and `BUILD`.
2. Ask for **JOB** and **BUILD** only if missing; do not ask "what next?" between stages.
3. On CLI errors (auth, network), stop and report; do not invent build data.
4. Use **`coreos-tools jenkins`** commands (Go CLI in the container).