Prow Job Analyze Test Failure

Prow Job Analyze Test Failure

This skill analyzes test failures by downloading Prow CI artifacts, checking test logs, inspecting resources and events, analyzing test source code, and optionally integrating cluster diagnostics from must-gather data.

Prerequisites

Identical with "Prow Job Analyze Resource" skill.

Input Format

The user will provide:

1. Prow job URL - gcsweb URL containing test-platform-results/

   • Example: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_hypershift/6731/pull-ci-openshift-hypershift-main-e2e-aws/1962527613477982208
   • URL may or may not have trailing slash

2. Test name (optional) - specific test name that failed

   • When provided, focus the analysis on that test's stack trace and logs
   • When omitted, analyze all failed CI steps by inspecting the JUnit XML, build logs, and step artifacts to identify the root cause — this is the common case for multi-step CI workflows (e.g. HyperShift, bare-metal OADP jobs) where the failure is in a step rather than a named unit test
   • Examples:
     • TestKarpenter/EnsureHostedCluster/ValidateMetricsAreExposed
     • TestCreateClusterCustomConfig
     • The openshift-console downloads pods [apigroup:console.openshift.io] should be scheduled on different nodes

3. Optional flags (optional):

   • --fast - Skip must-gather extraction and analysis; continue with log/JUnit/step-artifact analysis only (scope is preserved: test-level when test-name is provided, step-level across all failed CI steps when omitted)

Implementation Steps

Step 1: Parse and Validate URL

Use the "Parse and Validate URL" steps from "Prow Job Analyze Resource" skill

Step 2: Create Working Directory

1. Check for existing artifacts first

   • Check if .work/prow-job-analyze-test-failure/{build_id}/logs/ directory exists and has content
   • If it exists with content:
     • Use AskUserQuestion tool to ask:
       • Question: "Artifacts already exist for build {build_id}. Would you like to use the existing download or re-download?"
       • Options:
         • "Use existing" - Skip to step Analyze Test Failure
         • "Re-download" - Continue to clean and re-download
     • If user chooses "Re-download":
       • Remove all existing content: rm -rf .work/prow-job-analyze-test-failure/{build_id}/logs/
       • Also remove tmp directory: rm -rf .work/prow-job-analyze-test-failure/{build_id}/tmp/
       • This ensures clean state before downloading new content
     • If user chooses "Use existing":
       • Skip directly to Step 4 (Analyze Test Failure)
       • Still need to download prowjob.json if it doesn't exist

2. Create directory structure

   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/tmp
   

   • Use .work/prow-job-analyze-test-failure/ as the base directory (already in .gitignore)
   • Use build_id as subdirectory name
   • Create logs/ subdirectory for all downloads
   • Create tmp/ subdirectory for temporary files (intermediate JSON, etc.)
   • Working directory: .work/prow-job-analyze-test-failure/{build_id}/

Step 3: Download and Validate prowjob.json

Use the fetch-prowjob-json skill to fetch the prowjob.json for this job. See plugins/ci/skills/fetch-prowjob-json/SKILL.md for complete implementation details.

1. Fetch prowjob.json using the Prow job URL (convert to gcsweb URL per the fetch-prowjob-json skill)
2. Save locally to .work/prow-job-analyze-test-failure/{build_id}/logs/prowjob.json
3. Parse and validate
   • Search for pattern: --target=([a-zA-Z0-9-]+) in the ci-operator args
   • If not found:
     • Display: "This is not a ci-operator job. The prowjob cannot be analyzed by this skill."
     • Explain: ci-operator jobs have a --target argument specifying the test target
     • Exit skill
4. Extract target name and JOB_NAME
   • Capture the target value (e.g., e2e-aws-ovn) from the --target= arg
   • Extract JOB_NAME from .spec.job in prowjob.json (the artifact directory key):

     JOB_NAME=$(jq -r '.spec.job' .work/prow-job-analyze-test-failure/{build_id}/logs/prowjob.json)
     

   • Note: on PR jobs {target} and {JOB_NAME} often differ. All artifact-path lookups (JUnit XML, step logs, step artifacts) must use {JOB_NAME}, not {target}

Step 4: Analyze Test Failure

Step 4.0: Detect Aggregated Jobs

Aggregated jobs run the same job in parallel (typically 10 times) and perform statistical analysis of test results. Detect aggregation by checking for aggregated- prefix in the job name or an aggregator container/step in prowjob.json.

If the job is aggregated, the failure modes are different from normal jobs:

1. Statistically significant test failure — The test itself fails frequently enough across runs to be flagged as a regression. This is a real regression that needs investigation.

2. Insufficient completed runs — Not enough runs completed successfully to perform the statistical test (e.g., only 5 of 10 jobs produced results). This manifests as mass test failures across many unrelated tests. The root cause is whatever prevented the other runs from completing — this could be infrastructure issues, install failures, or an actual product bug causing crashes. Investigate the underlying job runs that did not complete to determine why.

3. Non-deterministic test presence — A test only ran in a small subset of completed jobs, even though the other jobs completed successfully. The failure message will say something like "Passed 1 times, failed 0 times, skipped 0 times: we require at least 6 attempts to have a chance at success". Every test must produce results in every job run; it is a bug if it does not. This is a regression — someone introduced a test that doesn't produce results deterministically. Investigate which test is non-deterministic and why it only runs in some jobs.

When analyzing an aggregated job failure, first determine which failure mode applies before diving into individual test analysis. For mode 2, focus on why runs failed rather than individual test results. For mode 3, investigate why the test only ran in some jobs.

Finding underlying job run URLs:

The aggregated junit XML contains links to every underlying job run. Download it from:

gs://test-platform-results/{bucket-path}/artifacts/release-analysis-aggregator/openshift-release-analysis-aggregator/artifacts/release-analysis-aggregator/{job-name}/{payload-tag}/junit-aggregated.xml

Each <testcase> element has a <system-out> with YAML-formatted data including passes:, failures:, and skips: lists. Each entry has:

• jobrunid: The build ID of the underlying job run
• humanurl: Prow URL for the job run (e.g., https://prow.ci.openshift.org/view/gs/test-platform-results/logs/{job-name}/{jobrunid})
• gcsartifacturl: Direct link to GCS artifacts

Use the humanurl links to investigate individual job run failures with the normal (non-aggregated) analysis steps below.

Step 4.1: Download build-log.txt

gcloud storage cp gs://test-platform-results/{bucket-path}/build-log.txt .work/prow-job-analyze-test-failure/{build_id}/logs/build-log.txt --no-user-output-enabled

Step 4.2: Parse and validate

• Read .work/prow-job-analyze-test-failure/{build_id}/logs/build-log.txt

Branch on whether {test_name} was provided:

Branch A: test_name provided

• Search build-log.txt for the exact test name string
• Gather the stack trace and surrounding context for that specific test
• Store the single failing test's error output for use in Steps 4.4 and 4.9

Branch B: test_name NOT provided (multi-step discovery)

This is the common case for multi-step CI workflows (e.g., HyperShift, bare-metal, OADP jobs) where failures occur in CI steps rather than named unit tests.

1. Discover failed CI steps from JUnit XML

   Use {JOB_NAME} (from Step 3, extracted from .spec.job) for all GCS artifact paths — not {target}. On PR jobs these values differ and {target} will miss the artifacts.

   Search for JUnit XML files under the artifacts directory:

   gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/**/junit*.xml" 2>/dev/null
   

   Download all found JUnit XML files:

   gcloud storage cp "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/**/junit*.xml" \
     .work/prow-job-analyze-test-failure/{build_id}/logs/ --no-user-output-enabled --recursive 2>/dev/null || true
   

   Parse each downloaded XML file and collect every <testcase> element where:

   • A <failure> or <error> child element is present, OR
   • The <testcase> has attribute status="failed"

   For each failed testcase record:

   • step_name: value of classname or name attribute (whichever identifies the CI step)
   • failure_message: text content of the <failure> or <error> element
   • junit_file: path of the XML file it came from

2. Classify each failed step by phase

   The ci-operator JUnit XML (junit_operator.xml) includes phase-level testcases:

   • "Run multi-stage test pre phase" — setup/installation steps
   • "Run multi-stage test test phase" — functional test steps
   • "Run multi-stage test post phase" — gather/cleanup steps

   Use the phase entries to classify each failed step_name:

   • Look at the failed phase-level testcases (those with "pre phase", "test phase", or "post phase" in their name) — these tell you which phase failed
   • Cross-reference the individual step testcases against their phase:
     • Steps in the pre phase → installation/setup steps
     • Steps in the test phase → functional test steps
     • Steps in the post phase → gather/cleanup steps (rarely the root cause)

   If the phase-level testcases are absent from the JUnit XML (some jobs omit them), fall back to the ci-operator-step-graph.json artifact, which lists all steps with their dependencies and timing — use execution order and naming conventions as a secondary signal. When still ambiguous, prefer classifying as a test step so must-gather analysis is not skipped.

3. Route each failed step to the appropriate analysis path

   • pre phase steps (installation) → invoke the ci:analyze-prow-job-install-failure skill for each such step, passing the same Prow job URL. Must-gather is not attempted for pre phase failures: must-gather requires a live apiserver, and when the installation phase fails the cluster is not fully up, so collection will fail. Capture the delegated output and store it as a per-step entry in the shared results collection used by Step 5 and Step 5.5:

     {
       step_name:      {step_name},
       type:           "installation",
       summary:        {one-line summary from ci:analyze-prow-job-install-failure},
       evidence:       {key error / stack trace from delegated analysis},
       recommendation: {recommended action from delegated analysis}
     }
     

   • test phase steps (functional tests) → proceed with the iteration in item 4 below (download step log, artifacts, extract stack trace) and include must-gather analysis (Steps 4.4–4.9). Store findings in the same per-step results collection:

     {
       step_name:      {step_name},
       type:           "test",
       summary:        {one-line failure summary},
       evidence:       {stack trace / error output},
       recommendation: {recommended action}
     }
     

   • post phase steps (gather/cleanup) → treat as informational. Download the step log and note the failure in the report, but do not perform must-gather analysis for post-phase failures alone (they are usually a consequence of earlier failures).

4. Iterate over each test phase step (pre and post phase steps already routed above)

   For each step_name in the test phase, perform the following sub-steps:

   a. Download step-specific log

   gcloud storage cp \
     "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/{step_name}/build-log.txt" \
     .work/prow-job-analyze-test-failure/{build_id}/logs/{step_name}-build-log.txt \
     --no-user-output-enabled 2>/dev/null || true
   

   If the step log is not found at that path, fall back to scanning build-log.txt for lines mentioning {step_name} and collect surrounding context (±50 lines).

   b. Download step-specific artifacts

   gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/{JOB_NAME}/{step_name}/" \
     2>/dev/null
   

   Download any relevant artifacts (e.g., *.json, *.yaml, events*.txt) to .work/prow-job-analyze-test-failure/{build_id}/logs/{step_name}/.

   c. Extract stack trace and context

   • Scan the step log for panic traces, FAIL, Error:, fatal, or assertion failures
   • Collect the full stack trace block (from the triggering line to the end of the trace)
   • Note the failure message from the JUnit XML as additional context

   d. Store per-step findings Record for each step:

   • step_name
   • failure_message (from JUnit XML)
   • stack_trace (from step log or build-log.txt context)
   • artifacts (list of downloaded artifact paths)

5. Produce a multi-step failure summary

   After iterating all failed steps, produce an ordered list of findings:

   Failed Steps Discovered:
   1. {step_name_1}: {one-line failure summary}
   2. {step_name_2}: {one-line failure summary}
   ...
   

   This list is used in Step 4.4 (evidence gathering) and Step 5 (report) to drive per-step analysis rather than singular test analysis.

Step 4.3: Examine intervals files for cluster activity during E2E failures

• Search recursively for E2E timeline artifacts (known as "interval files") within the bucket-path:

  gcloud storage ls 'gs://test-platform-results/{bucket-path}/**/e2e-timelines_spyglass_*json'
  

• The files can be nested at unpredictable levels below the bucket-path
• There could be as many as two matching files
• Download all matching interval files (use the full paths from the search results):

  gcloud storage cp gs://test-platform-results/{bucket-path}/**/e2e-timelines_spyglass_*.json .work/prow-job-analyze-test-failure/{build_id}/logs/ --no-user-output-enabled
  

• If the wildcard copy doesn't work, copy each file individually using the full paths from the search results
• Scan interval files for test failure timing:
  • Look for intervals where source = "E2ETest" and message.annotations.status = "Failed"
  • Note the from and to timestamps on this interval - this indicates when the test was running
• Scan interval files for related cluster events:
  • Look for intervals that overlap the timeframe when the failed test was running
  • Filter for intervals with:
    • level = "Error" or level = "Warning"
    • source = "OperatorState"
  • These events may indicate cluster issues that caused or contributed to the test failure

Step 4.3b: Check for Known Symptom Labels

The CI system may attach symptom labels to job runs — machine-detected patterns (e.g., "test failures during high CPU events") stored as JSON artifacts. These are not root causes but provide useful environmental context that may help explain failures when no other cause is found.

1. List the job_labels directory

   gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/job_labels/" 2>/dev/null
   

   • If the directory does not exist or returns an error, skip this step silently

2. Download any JSON symptom files (exclude label-summary.html)

   gcloud storage cp "gs://test-platform-results/{bucket-path}/artifacts/job_labels/*.json" \
     .work/prow-job-analyze-test-failure/{build_id}/logs/job_labels/ --no-user-output-enabled 2>/dev/null || true
   

3. Parse symptom labels — each JSON file describes a detected symptom with a summary and explanation. Collect all symptom summaries for inclusion in the report.

4. Use symptoms as investigative context — symptoms are environmental observations, NOT definitive causes. They should inform your investigation (e.g., if 2 tests failed while high CPU was measured, CPU pressure could explain the failures if no other cause is found) but you must still perform thorough root cause analysis. Include them in the "Known Symptoms Seen" section of the report.

Step 4.4: Gather initial evidence

• Analyze stack traces from build-log.txt
• Analyze related code in the code repository
• Store artifacts from Prow CI job (json/yaml files) related to the failure under .work/prow-job-analyze-test-failure/{build_id}/tmp
• Store logs under .work/prow-job-analyze-test-failure/{build_id}/logs/
• Collect evidence from logs and events and other json/yaml files
• Check for CI step script errors: If the error is a scripting issue in a CI step (e.g., unbound variable, syntax error, missing command, bad exit code from a shell script) rather than a product bug, check for recent commits to that step's script in the openshift/release repository. This is part of evidence gathering — identifying the responsible PR early informs the rest of the analysis. Include the responsible PR in your evidence.

Step 4.4b: Investigate crash-looping or failing containers

Be tenacious. When the build log or test output mentions crash-looping pods, container restarts, or deployments not becoming ready, you MUST trace the failure to its root cause. Never stop at "containers are crash-looping" — find out why.

Pursue all available log sources:

• Must-gather data (Step 4.6–4.7): Contains pod YAMLs with containerStatuses (exitCode, lastState.terminated.reason, restartCount), container logs (current and previous), events, and operator conditions. This is often the richest source for diagnosing crash-looping pods.
• Gather-extra / gather-audit logs: Some jobs run additional gather steps that collect extra diagnostics. List the step artifacts directory for gather steps beyond gather-must-gather (e.g., gather-extra, gather-audit-logs) and download relevant logs.
• Step-level build logs: The build log for the failing test step (downloaded in Step 4.2) often contains error output, stack traces, and timeout messages that reference specific pods or containers.
• Events from interval files (Step 4.3): Operator state transitions and warning events correlated with the failure window.

Follow the dependency chain. Always trace upstream to the originating error rather than stopping at the first symptom.

Step 4.5: Check for Must-Gather Availability

1. Parse optional flags

   • Parse user input for --fast flag
   • If --fast flag present:
     • Skip must-gather detection and analysis entirely
     • Proceed directly to Step 5 — scope is preserved:
       • If test_name was provided → produce singular test report (Branch A)
       • If test_name was omitted → produce multi-step report (Branch B)
     • Do NOT prompt user about must-gather

2. Extract actual test name from prowjob.json

   The artifacts directory uses the test name from prowjob.json, NOT the full URL path.

   # Extract test name from prowjob.json (e.g., "e2e-aws-operator-serial-ote")
   JOB_NAME=$(jq -r '.spec.job' .work/prow-job-analyze-test-failure/{build_id}/logs/prowjob.json)
   
   # Note: For PR jobs, TARGET contains the full PR path like:
   #   pr-logs/pull/openshift_service-ca-operator/306/pull-ci-openshift-service-ca-operator-main-e2e-aws-operator-serial-ote
   # But artifacts are stored under just the test name:
   #   e2e-aws-operator-serial-ote
   

3. Detect must-gather archive (only if --fast not present)

   Use JOB_NAME (not TARGET) for artifact paths.

   HyperShift jobs may have different must-gather patterns:

   Pattern 1: Unified Archive (dump-management-cluster)

   • Single archive with both management and hosted cluster data
   • Used by: hypershift-aws-e2e-external workflow

   Pattern 2: Dual Archives (gather-must-gather + dump)

   • Standard must-gather for management cluster
   • Separate hypershift-dump for additional data (may or may not have hosted cluster)
   • Used by: hypershift-kubevirt-e2e-aws workflow

   Pattern 3: Standard Only (gather-must-gather)

   • Standard OpenShift must-gather only
   • No HyperShift-specific dump

   Detection logic:

   # Check for Pattern 1: Unified archive (dump-management-cluster)
   UNIFIED_DUMP=$(gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/$JOB_NAME/dump-management-cluster/artifacts/artifacts.tar*" 2>/dev/null | head -1 || true)
   
   # Check for Pattern 2/3: Standard must-gather
   STANDARD_MG=$(gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/$JOB_NAME/gather-must-gather/artifacts/must-gather.tar" 2>/dev/null || true)
   
   # Check for Pattern 2: Additional hypershift-dump (multiple possible locations)
   # Use wildcards to match all current and future HyperShift dump patterns:
   # 1. **/artifacts/hypershift-dump.tar (covers dump/, hypershift-mce-dump/, etc.)
   # 2. **/artifacts/**/hostedcluster.tar (covers all E2E test patterns)
   HYPERSHIFT_DUMP=""
   for pattern in \
       "**/artifacts/hypershift-dump.tar" \
       "**/artifacts/**/hostedcluster.tar"; do
       FOUND=$(gcloud storage ls "gs://test-platform-results/{bucket-path}/artifacts/$JOB_NAME/$pattern" 2>/dev/null | head -1 || true)
       if [ -n "$FOUND" ]; then
           HYPERSHIFT_DUMP="$FOUND"
           break
       fi
   done
   
   # Determine pattern and check for hosted cluster data
   if [ -n "$UNIFIED_DUMP" ]; then
     # Pattern 1: Unified archive
     PATTERN="unified"
   
     # Download temporarily to check for hosted cluster data
     TMP_CHECK="/tmp/check-unified-$$.tar"
     gcloud storage cp "$UNIFIED_DUMP" "$TMP_CHECK" --no-user-output-enabled
   
     # Check if archive contains hostedcluster-* directory
     HAS_HOSTED_CLUSTER=$(tar -tf "$TMP_CHECK" 2>/dev/null | grep -q "hostedcluster-" && echo "true" || echo "false")
     rm -f "$TMP_CHECK"
   
   elif [ -n "$STANDARD_MG" ] && [ -n "$HYPERSHIFT_DUMP" ]; then
     # Pattern 2: Dual archives
     PATTERN="dual"
   
     # Download hypershift-dump temporarily to check for hosted cluster
     TMP_CHECK="/tmp/check-dump-$$.tar"
     gcloud storage cp "$HYPERSHIFT_DUMP" "$TMP_CHECK" --no-user-output-enabled
   
     # Check if hypershift-dump contains hostedcluster-* directory
     HAS_HOSTED_CLUSTER=$(tar -tf "$TMP_CHECK" 2>/dev/null | grep -q "hostedcluster-" && echo "true" || echo "false")
     rm -f "$TMP_CHECK"
   
   elif [ -n "$STANDARD_MG" ]; then
     # Pattern 3: Standard must-gather only
     PATTERN="standard"
     HAS_HOSTED_CLUSTER=false
   
   else
     # No must-gather found
     PATTERN="none"
     HAS_HOSTED_CLUSTER=false
   fi
   

   Possible outcomes:

   • PATTERN="none": No must-gather found → Skip to Step 5 (silent, expected for some jobs)
   • PATTERN="standard": Standard OpenShift must-gather only → Extract single cluster
   • PATTERN="unified" + HAS_HOSTED_CLUSTER=false: HyperShift unified archive with management only
   • PATTERN="unified" + HAS_HOSTED_CLUSTER=true: HyperShift unified archive with both clusters
   • PATTERN="dual" + HAS_HOSTED_CLUSTER=false: Two archives but hosted cluster not in hypershift-dump
   • PATTERN="dual" + HAS_HOSTED_CLUSTER=true: Two archives with hosted cluster in hypershift-dump

   Important Pattern Notes:

   • Pattern 1 (unified): Management at logs/artifacts/output/, hosted at logs/artifacts/output/hostedcluster-{name}/
   • Pattern 2 (dual): Management in standard must-gather, hosted MAY be in hypershift-dump.tar or hostedcluster.tar (not guaranteed)
     • Wildcard patterns match all current and future dump locations
   • Pattern 3 (standard): Management cluster only, no HyperShift-specific data

4. Ask user if they want must-gather analysis

   • Only if must-gather(s) were found and --fast not present
   • Use AskUserQuestion tool:
     • Question: "Must-gather data is available. Include cluster diagnostics in the analysis?"
     • Header: "Must-gather"
     • Options:
       • Label: "Yes - Extract and analyze must-gather (Recommended)" Description: "Provides cluster-level diagnostics that may reveal root causes (pods, operators, nodes, events). Takes additional time to download and analyze."
       • Label: "No - Skip must-gather (faster)" Description: "Only analyze test-level artifacts (build-log, intervals). Faster but may miss cluster-level issues."
   • If user chooses "No", skip to Step 5

Step 4.6: Extract Must-Gather (Conditional)

Only if user chose "Yes" in Step 4.5:

1. Determine extraction strategy

   • If single must-gather detected → extract to must-gather/logs/
   • If dual must-gather detected (HyperShift) → extract both:
     • Management cluster → must-gather-mgmt/logs/
     • Hosted cluster → must-gather-hosted/logs/

2. Check for existing extraction

   For single must-gather:

   • Check if .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/ exists with content

   For dual must-gather (HyperShift):

   • Check if .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/ exists with content
   • Check if .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/ exists with content

   If either exists:

   • Use AskUserQuestion tool:
     • Question: "Must-gather already extracted for this build. Use existing data?"
     • Header: "Reuse"
     • Options:
       • Label: "Use existing" Description: "Reuse previously extracted must-gather data (faster)"
       • Label: "Re-extract" Description: "Download and extract fresh must-gather data"
   • If "Re-extract":
     • rm -rf .work/prow-job-analyze-test-failure/{build_id}/must-gather*/
     • Continue to step 3 (fresh extraction)
   • If "Use existing":
     • Validate content directories exist and are not empty (see validation in step 4.6)
     • If validation fails, fall back to re-extraction
     • If validation succeeds, skip to Step 4.7

3. Create must-gather directories

   Based on PATTERN from Step 4.5.3:

   For Pattern 3 (standard only):

   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather/tmp
   

   For Pattern 1 (unified) or Pattern 2 (dual):

   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs
   mkdir -p .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp
   

4. Download must-gather archives

   Use JOB_NAME (from Step 4.5.2) for artifact paths, not {target}:

   For Pattern 3 (standard only):

   gcloud storage cp "$STANDARD_MG" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather/tmp/must-gather.tar \
     --no-user-output-enabled
   

   For Pattern 1 (unified):

   # Download unified archive (contains both management and hosted cluster data)
   gcloud storage cp "$UNIFIED_DUMP" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/unified-dump.tar \
     --no-user-output-enabled
   

   For Pattern 2 (dual):

   # Download management cluster must-gather
   gcloud storage cp "$STANDARD_MG" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/must-gather.tar \
     --no-user-output-enabled
   
   # Download hypershift-dump (may or may not contain hosted cluster)
   gcloud storage cp "$HYPERSHIFT_DUMP" \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp/hypershift-dump.tar \
     --no-user-output-enabled
   

5. Extract archives

   For Pattern 3 (standard only):

   # Use existing extract_archives.py script for standard must-gather
   python3 plugins/ci/skills/prow-job-extract-must-gather/extract_archives.py \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather/tmp/must-gather.tar \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs
   

   For Pattern 1 (unified):

   # Extract unified archive to temporary location
   TMP_EXTRACT=".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/extracted"
   mkdir -p "$TMP_EXTRACT"
   
   # Handle both .tar and .tar.gz
   if [[ "$UNIFIED_DUMP" == *.tar.gz ]]; then
     tar -xzf .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/unified-dump.tar -C "$TMP_EXTRACT"
   else
     tar -xf .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/unified-dump.tar -C "$TMP_EXTRACT"
   fi
   
   # Find the output directory (may be at logs/artifacts/output or just output)
   OUTPUT_DIR=$(find "$TMP_EXTRACT" -type d -name "output" | head -1)
   
   if [ -z "$OUTPUT_DIR" ]; then
     echo "ERROR: Could not find output directory in unified dump"
     rm -rf "$TMP_EXTRACT"
     # Clear variables to prevent subsequent usage
     HAS_HOSTED_CLUSTER="false"
     unset HOSTED_DIR
     unset OUTPUT_DIR
     # Skip to Step 5 - no must-gather analysis possible
   else
     # Move management cluster data (root level in output/)
     # Exclude hostedcluster-* directories
     for item in "$OUTPUT_DIR"/*; do
       if [ -e "$item" ] && [[ ! "$(basename "$item")" =~ ^hostedcluster- ]]; then
         mv "$item" .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/
       fi
     done
   
     # Move hosted cluster data (hostedcluster-* subdirectory)
     if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
       HOSTED_DIR=$(find "$OUTPUT_DIR" -maxdepth 1 -type d -name "hostedcluster-*" | head -1)
       if [ -n "$HOSTED_DIR" ]; then
         mv "$HOSTED_DIR"/* .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/
         echo "✓ Hosted cluster data extracted from unified archive"
       else
         echo "WARNING: Expected hosted cluster data but hostedcluster-* directory not found"
       fi
     fi
   
     # Cleanup temporary extraction directory
     rm -rf "$TMP_EXTRACT"
   fi
   

   For Pattern 2 (dual):

   # Extract management cluster must-gather (standard format)
   python3 plugins/prow-job/skills/prow-job-extract-must-gather/extract_archives.py \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/tmp/must-gather.tar \
     .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs
   
   # Extract hypershift-dump (may contain hosted cluster)
   TMP_EXTRACT=".work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp/extracted"
   mkdir -p "$TMP_EXTRACT"
   tar -xf .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/tmp/hypershift-dump.tar -C "$TMP_EXTRACT"
   
   if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
     # Look for hostedcluster-* directory in dump
     HOSTED_DIR=$(find "$TMP_EXTRACT" -maxdepth 2 -type d -name "hostedcluster-*" | head -1)
     if [ -n "$HOSTED_DIR" ]; then
       mv "$HOSTED_DIR"/* .work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs/
       echo "✓ Hosted cluster data extracted from hypershift-dump"
     else
       echo "WARNING: HAS_HOSTED_CLUSTER=true but no hostedcluster-* directory found in dump"
       echo "Hypershift-dump likely contains only management cluster data"
       HAS_HOSTED_CLUSTER=false  # Update flag since hosted cluster not actually present
     fi
   else
     echo "INFO: Hypershift-dump does not contain hosted cluster data (management cluster only)"
   fi
   
   # Cleanup temporary extraction directory
   rm -rf "$TMP_EXTRACT"
   

6. Locate and validate content directories

   For Pattern 3 (standard only):

   # Check for content/ directory first (renamed by extraction script)
   if [ -d ".work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/content" ]; then
       MUST_GATHER_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather/logs/content"
   else
       # Fall back to finding the directory containing -ci- (e.g., registry-build09-ci-...)
       MUST_GATHER_PATH=$(find .work/prow-job-analyze-test-failure/{build_id}/must-gather/logs -maxdepth 1 -type d -name "*-ci-*" | head -1)
   fi
   
   # Validate MUST_GATHER_PATH is set and directory exists
   if [ -z "$MUST_GATHER_PATH" ] || [ ! -d "$MUST_GATHER_PATH" ]; then
       echo "ERROR: Must-gather content directory not found after extraction"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   elif [ -z "$(ls -A "$MUST_GATHER_PATH" 2>/dev/null)" ]; then
       echo "ERROR: Must-gather content directory is empty"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   else
       echo "✓ Must-gather content located at: $MUST_GATHER_PATH"
       # Continue to Step 4.7 with MUST_GATHER_PATH set
   fi
   

   For Pattern 1 (unified) or Pattern 2 (dual):

   # Management cluster validation
   if [ "$PATTERN" = "unified" ]; then
       # Pattern 1: Data extracted directly to logs/ directory
       MUST_GATHER_MGMT_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs"
   else
       # Pattern 2: Standard must-gather extraction (look for content/ or hash directory)
       if [ -d ".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/content" ]; then
           MUST_GATHER_MGMT_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs/content"
       else
           MUST_GATHER_MGMT_PATH=$(find .work/prow-job-analyze-test-failure/{build_id}/must-gather-mgmt/logs -maxdepth 1 -type d -name "*-ci-*" | head -1)
       fi
   fi
   
   # Validate management cluster path
   if [ -z "$MUST_GATHER_MGMT_PATH" ] || [ ! -d "$MUST_GATHER_MGMT_PATH" ]; then
       echo "ERROR: Management cluster directory not found"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   elif [ -z "$(ls -A "$MUST_GATHER_MGMT_PATH" 2>/dev/null)" ]; then
       echo "ERROR: Management cluster directory is empty"
       # Skip to Step 5 (scope preserved: Branch A or Branch B per test_name presence)
   else
       echo "✓ Management cluster data located at: $MUST_GATHER_MGMT_PATH"
   fi
   
   # Hosted cluster validation - only if HAS_HOSTED_CLUSTER is true
   if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
       MUST_GATHER_HOSTED_PATH=".work/prow-job-analyze-test-failure/{build_id}/must-gather-hosted/logs"
   
       # Validate hosted cluster path
       if [ ! -d "$MUST_GATHER_HOSTED_PATH" ]; then
           echo "WARNING: Hosted cluster directory not found (expected based on archive detection)"
           MUST_GATHER_HOSTED_PATH=""  # Clear the path
       elif [ -z "$(ls -A "$MUST_GATHER_HOSTED_PATH" 2>/dev/null)" ]; then
           echo "WARNING: Hosted cluster directory is empty"
           MUST_GATHER_HOSTED_PATH=""  # Clear the path
       else
           echo "✓ Hosted cluster data located at: $MUST_GATHER_HOSTED_PATH"
       fi
   else
       echo "✓ Management cluster must-gather located at: $MUST_GATHER_MGMT_PATH"
   fi
   
   # Only validate hosted cluster if HAS_HOSTED_CLUSTER is true
   if [ "$HAS_HOSTED_CLUSTER" = "true" ]; then
       if [ -z "$MUST_GATHER_HOSTED_PATH" ] || [ ! -d "$MUST_GATHER_HOSTED_PATH" ]; then
           echo "ERROR: Hosted cluster must-gather content directory not found"
       elif [ -z "$(ls -A "$MUST_GATHER_HOSTED_PATH" 2>/dev/null)" ]; then
           echo "ERROR: Hosted cluster must-gather content directory is empty"
       else
           echo "✓ Hosted cluster must-gather located at: $MUST_GATHER_HOSTED_PATH"
           echo "✓ Hosted cluster namespace: $HOSTED_NAMESPACE"
       fi
   fi
   

Step 4.7: Analyze Must-Gather (Conditional)

Only if Step 4.6 completed successfully:

1. Locate must-gather-analyzer scripts

   The must-gather plugin provides analysis scripts. Locate the scripts directory:

   # Try to find the must-gather-analyzer scripts in common locations
   for SEARCH_PATH in \
       "plugins/must-gather/skills/must-gather-analyzer/scripts" \
       "~/.claude/plugins/cache/*/plugins/must-gather/skills/must-gather-analyzer/scripts" \
       "$(find ~ -type d -path "*/must-gather/skills/must-gather-analyzer/scripts" 2>/dev/null | head -1)"; do
       SCRIPTS_DIR=$(eval echo "$SEARCH_PATH")
       if [ -d "$SCRIPTS_DIR" ] && [ -f "$SCRIPTS_DIR/analyze_clusteroperators.py" ]; then
           break
       fi
       SCRIPTS_DIR=""
   done
   
   if [ -z "$SCRIPTS_DIR" ]; then
       echo "WARNING: Must-gather analysis scripts not found."
       echo "Install the must-gather plugin: /plugin install must-gather@ai-helpers"
       # Continue to Step 5 without cluster analysis
   fi
   

2. Run targeted cluster diagnostics

   Focus on issues relevant to test failures (not full cluster analysis).

   For single must-gather (standard OpenShift):

   # Core diagnostics (only if scripts and path are available)
   if [ -n "$SCRIPTS_DIR" ] && [ -n "$MUST_GATHER_PATH" ]; then
       python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" "$MUST_GATHER_PATH"
       python3 "$SCRIPTS_DIR/analyze_pods.py" "$MUST_GATHER_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_nodes.py" "$MUST_GATHER_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_events.py" "$MUST_GATHER_PATH" --type Warning --count 50
   else
       echo "WARNING: Skipping must-gather analysis (scripts or path not available)"
   fi
   

   For dual must-gather (HyperShift):

   # Management cluster diagnostics (only if scripts and path are available)
   if [ -n "$SCRIPTS_DIR" ] && [ -n "$MUST_GATHER_MGMT_PATH" ]; then
       echo "=== Analyzing Management Cluster ==="
       python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" "$MUST_GATHER_MGMT_PATH"
       python3 "$SCRIPTS_DIR/analyze_pods.py" "$MUST_GATHER_MGMT_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_nodes.py" "$MUST_GATHER_MGMT_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_events.py" "$MUST_GATHER_MGMT_PATH" --type Warning --count 50
   else
       echo "WARNING: Skipping management cluster analysis (scripts or path not available)"
   fi
   
   # Hosted cluster diagnostics (only if scripts and path are available)
   if [ -n "$SCRIPTS_DIR" ] && [ -n "$MUST_GATHER_HOSTED_PATH" ]; then
       echo "=== Analyzing Hosted Cluster (Namespace: $HOSTED_NAMESPACE) ==="
       python3 "$SCRIPTS_DIR/analyze_clusteroperators.py" "$MUST_GATHER_HOSTED_PATH"
       python3 "$SCRIPTS_DIR/analyze_pods.py" "$MUST_GATHER_HOSTED_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_nodes.py" "$MUST_GATHER_HOSTED_PATH" --problems-only
       python3 "$SCRIPTS_DIR/analyze_events.py" "$MUST_GATHER_HOSTED_PATH" --type Warning --count 50
   else
       echo "INFO: Skipping hosted cluster analysis (scripts or path not available)"
   fi
   

3. Run conditional diagnostics based on test context

   # Network diagnostics (if test name suggests network issues)
   if [[ "$JOB_NAME" =~ network|ovn|sdn|connectivity|route|ingress|egress ]]; then
       if [ -n "$MUST_GATHER_PATH" ]; then
           python3 "$SCRIPTS_DIR/analyze_network.py" "$MUST_GATHER_PATH"
       fi
       if [ -n "$MUST_GATHER_MGMT_PATH" ]; then
   

…(truncated)