Prow Job Analysis

Prow Job Analysis

Analyze failures in OpenShift Prow CI jobs. Identify the job type, inspect artifacts, classify the failure, and route to the specialized reference for deep analysis.

Input Format

The user will provide:

1. Prow job URL (required) — Prow UI or gcsweb URL

   • https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job>/<build_id>
   • https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/...

2. Test name (optional) — specific failed test to focus on

3. Flags (optional):

   • --backends <list> — focus disruption analysis on specific backends

Prerequisites

• Python 3.7+: which python3
• jq: which jq
• gcloud CLI (recommended, not required): which gcloud — fastest access to the public bucket (no auth required). Without it, every artifact operation works over plain HTTPS: prow_job_artifact_search.py (stdlib-only list/search/fetch) or curl against https://storage.googleapis.com/test-platform-results/....

Investigation Workflow

Step 1: Parse URL and Extract Metadata

1. Find test-platform-results/ in the URL and extract the bucket path
2. Extract build_id — pattern (\d{10,}) in the path
3. Construct GCS base: gs://test-platform-results/{bucket-path}/

Step 2: Fetch prowjob.json

Use the fetch-prowjob-json skill to get job metadata. Extract:

• Job name from .spec.job
• Target from --target= in ci-operator args
• Job state from .status.state
• Refs (org, repo, PR number) from .spec.refs

Step 3: Classify Job Type from Name

Parse the job name to determine the environment and expected failure modes:

Step 4: Download Key Artifacts

mkdir -p .work/prow-job-analysis/{build_id}/logs

# Build log (always)
gcloud storage cp gs://test-platform-results/{bucket-path}/build-log.txt \
  .work/prow-job-analysis/{build_id}/logs/ --no-user-output-enabled

# JUnit XML (always — identifies failed tests/steps)
gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/**/junit*.xml" 2>/dev/null

# Node journals (always, when the job created a cluster) — required input for the
# Step 5 OS-layer check. Gzip-compressed WITHOUT a .gz extension: zcat/zgrep only.
gcloud storage cp -r \
  "gs://test-platform-results/{bucket-path}/artifacts/{target}/gather-extra/artifacts/nodes" \
  .work/prow-job-analysis/{build_id}/ --no-user-output-enabled 2>/dev/null || true

Step 5: Classify Failure and Route to Reference

Examine the build log and JUnit results to classify the failure, then consult the appropriate reference file for detailed analysis procedures.

OS-layer evidence check (mandatory for every job, before routing)

Operating-system (RHCOS) layer breakage frequently masquerades as an unrelated product failure: a single RHCOS bump swaps the kernel, cri-o, systemd, NetworkManager, and SELinux policy across the whole cluster at once, so the real cause surfaces as a symptom in some other domain. Before selecting a row from the routing table, complete BOTH steps:

1. Compare runtime versions across boots in the node journals (downloaded in Step 4; gzip-compressed without a .gz extension — plain grep silently matches nothing, use zcat/zgrep):

# Runtime versions per boot. End-of-run snapshots (oc_cmds/nodes, nodes.json)
# show only the final version; changes within the run are visible only here.
zgrep -hE "Starting CRI-O, version|Container runtime initialized" \
  .work/prow-job-analysis/{build_id}/nodes/*/journal | sort | uniq -c

2. Scan the build log, JUnit, oc_cmds (node / clusteroperator status), MachineConfig data, and the journals for these signals:

• NetworkPluginNotReady, or a missing CNI config (/etc/cni/net.d empty / no CNI plugin)
• A ContainerRuntimeVersion change on nodes (cri-o version bump between runs)
• A MachineConfigDaemon (MCD) rendered-config diff touching passwd, files, or units
• Multiple nodes going NotReady after a reboot
• CreateContainerError, RunContainerError, or OCI runtime errors (crun / runc)
• Kernel panic, BUG, Oops, or soft lockup in node journals or the serial console
• avc: denied / SELinux denials
• The same failure spanning multiple unrelated jobs at a payload boundary

If step 1 shows more than one runtime version on any node, or any step-2 signal is present, the RHCOS layer is implicated: still route via the table below using whichever reference matches the surface symptom, but also read operating-system-changes.md alongside it. Never clear the OS layer from end-of-run snapshots alone.

Failure Routing Table

Job-name routing (Step 3) picks which reference to read. Failure classification (install | test | upgrade | infra) follows the root cause, not the job name.

Common Artifact Paths

These are the most frequently needed artifacts. See artifacts reference for the complete directory structure.

URL Formats

Both formats are accepted and interchangeable:

# Prow UI
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/{job}/{build_id}

# gcsweb (direct GCS browser)
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/{job}/{build_id}

The GCS bucket is always test-platform-results, publicly accessible, no auth required.

Tips

• Start with build-log.txt — it shows the ci-operator orchestration and which steps failed
• JUnit XML is the source of truth for test pass/fail status
• Job name encodes environment — always parse it before diving into logs
• Check prowjob.json for timing, payload tag, and whether the job timed out
• Upgrade jobs install first — an "upgrade" job failing at install is an install failure, not an upgrade failure
• Aggregated jobs need statistical analysis, not individual test debugging
• Use .work/prow-job-analysis/{build_id}/ as the working directory for downloads