# Debug E2e Tests > Debug failed Konflux e2e test runs by downloading logs and artifacts from GitHub Actions or OpenShift CI (Prow), analyzing test failures, and suggesting fixes. Use when the user asks to debug, investigate, or fix a failing e2e test, CI failure, or flaky test. Use when this capability is needed. - Skill: `tomevault-io/debug-e2e-tests` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/debug-e2e-tests` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/debug-e2e-tests/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/debug-e2e-tests --- # Debug E2E Tests Debug failed e2e test runs from **GitHub Actions** or **OpenShift CI (Prow)**. Downloads test logs and cluster artifacts, identifies root causes, and suggests fixes to this repo or upstream dependencies. **When this skill is activated, tell the user:** "Using the **debug-e2e-tests** skill to investigate this failure." Then show the checklist from the Workflow section below so the user can track progress. ## Prerequisites - `gh` CLI authenticated with access to `konflux-ci/konflux-ci` - `curl` available (for Prow artifact downloads) - `tar` and `unzip` available (standard on Linux/macOS) ## Workflow Copy this checklist and track progress: ``` - [ ] Step 1: Identify the CI system and failed run - [ ] Step 2: Download artifacts - [ ] Step 3: Analyze test output (JUnit + Ginkgo logs) - [ ] Step 4: Analyze cluster logs - [ ] Step 5: (Optional) Inspect the tested commit - [ ] Step 6: Determine root cause and suggest fix ``` ### Step 1: Identify the CI System and Failed Run First, determine which CI system the failure is from: | Signal | CI System | |--------|-----------| | User provides a GitHub Actions run ID or PR number | **GitHub Actions** | | URL contains `prow.ci.openshift.org` | **OpenShift CI (Prow)** | | URL contains `gcsweb-ci.apps.ci` | **OpenShift CI (Prow)** | | Job name starts with `pull-ci-` or `periodic-ci-` | **OpenShift CI (Prow)** | | User mentions "Prow", "OpenShift CI", or "OCP e2e" | **OpenShift CI (Prow)** | #### GitHub Actions If the user provides a PR number, run ID, or branch name, use that. Otherwise, find recent failures: ```bash # List recent failed e2e runs gh run list --repo konflux-ci/konflux-ci \ --workflow "Operator E2E Tests" \ --status failure --limit 10 # For a specific PR gh run list --repo konflux-ci/konflux-ci \ --workflow "Operator E2E Tests" \ --branch --limit 5 ``` Get the **run ID** from the output (first column). To see which jobs failed within a run: ```bash gh run view --repo konflux-ci/konflux-ci ``` The e2e workflow has these jobs: - **Run Operator E2E Tests (AMD64)** — main e2e on amd64 - **Run Operator E2E Tests (ARM64)** — main e2e on arm64 #### OpenShift CI (Prow) The user will typically provide a Prow URL like: ``` https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/konflux-ci_konflux-ci/// ``` Key Prow jobs for this repo: - `pull-ci-konflux-ci-konflux-ci-main-konflux-e2e-v420-optional` — PR presubmit (OCP 4.20) - `periodic-ci-konflux-ci-konflux-ci-main-ocp420-konflux-e2e-v420` — periodic x86_64 - `periodic-ci-konflux-ci-konflux-ci-main-ocp420-arm64-konflux-e2e-v420-arm64` — periodic ARM64 You can view recent Prow job runs at: - [x86_64 periodic](https://prow.ci.openshift.org/?job=periodic-ci-konflux-ci-konflux-ci-main-ocp420-konflux-e2e-v420) - [ARM64 periodic](https://prow.ci.openshift.org/?job=periodic-ci-konflux-ci-konflux-ci-main-ocp420-arm64-konflux-e2e-v420-arm64) ### Step 2: Download Artifacts #### GitHub Actions Use the helper script to download and extract artifacts: ```bash bash skills/debug-e2e-tests/scripts/download-logs.sh [arch] ``` - `arch` defaults to `amd64`. Use `arm64` if that's the failing job. - Artifacts are extracted to `/tmp/e2e-debug-/` If the script is unavailable, download manually: ```bash WORKDIR="/tmp/e2e-debug-" mkdir -p "$WORKDIR" && cd "$WORKDIR" # Download artifacts (name is "logs-amd64" or "logs-arm64") gh run download --repo konflux-ci/konflux-ci \ --name logs-amd64 --dir "$WORKDIR/logs" ``` Also download the raw job log for the failing job: ```bash # Get job IDs gh api "repos/konflux-ci/konflux-ci/actions/runs//jobs" \ --jq '.jobs[] | select(.conclusion=="failure") | "\(.id) \(.name)"' # Download the full job log gh api "repos/konflux-ci/konflux-ci/actions/jobs//logs" > "$WORKDIR/job.log" ``` #### OpenShift CI (Prow) Use the Prow download script: ```bash bash skills/debug-e2e-tests/scripts/download-prow-logs.sh ``` The script accepts: - A full Prow UI URL: `https://prow.ci.openshift.org/view/gs/...` - A GCS path: `gs://test-platform-results/pr-logs/...` - A gcsweb URL: `https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/...` Artifacts are extracted to `/tmp/prow-debug-/`. See [prow-reference.md](prow-reference.md) for the full artifact directory layout, step descriptions, gather file listing, and gcsweb URL patterns for fetching additional files. ### Step 3: Analyze Test Output #### 3a. JUnit Report Check the JUnit XML first for a quick summary of which tests failed: ```bash # GitHub Actions: # logs/junit/junit-conformance-amd64.xml (or arm64) # # Prow: # junit/junit.xml # (also: artifacts/junit_operator.xml for step-level pass/fail) ``` Look for `` elements with `` children. Extract test names and failure messages. #### 3b. Ginkgo Output The test execution log contains full Ginkgo output. Search for: - `[FAILED]` — Ginkgo failure markers - `FAIL!` — Go test failure - `Timed out` or `context deadline exceeded` — timeout failures - `Expected` / `to equal` / `to succeed` — Gomega assertion failures Where to find it: - **GitHub Actions:** `job.log` (raw job log) - **Prow:** `steps/konflux-ci-e2e-tests/build-log.txt` The test suite is `Konflux Conformance` under `test/go-tests/tests/conformance/`. Cross-reference the failing test name with the source files there to understand what the test was doing. ### Step 4: Analyze Cluster Logs #### GitHub Actions artifacts The `logs/` directory (from `generate-err-logs.sh`) contains the cluster state at test failure time. Read files in this priority order: | Priority | File | What to look for | |----------|------|------------------| | 1 | `operator-logs.log` | Operator crashes, reconcile errors, failed deployments | | 2 | `konflux-crs-status.log` | Konflux / sub-CR status conditions showing `Ready=False` | | 3 | `failed-pods-logs.log` | Container logs from pods with Warning events | | 4 | `failed-pods-definitions.yaml` | Pod specs showing image pull errors, resource limits, crash loops | | 5 | `failed-deployment-event-log.log` | Deployment rollout failures | | 6 | `pipelinerun-res.log` | PipelineRun status, failed tasks, error messages | | 7 | `taskrun-res.log` | TaskRun status, step container failures | | 8 | `cluster-resources.log` | Node pressure, pending pods, resource exhaustion | | 9 | `system-resources.log` | Host OOM, CPU saturation, disk full | For deeper analysis, check structured artifacts under `logs/artifacts/`: - `events.json` — all cluster events (search for Warning/error patterns) - `deployments.json` — deployment availability and conditions - `pipelineruns.json` / `taskruns.json` — Tekton resource details - `pods/_.log` — individual pod logs from key namespaces - `repositories.json` — PipelinesAsCode Repository CRs and their status For details on log structure, see [reference.md](reference.md). #### Prow artifacts For Prow jobs, cluster state is collected by the `redhat-appstudio-gather` step. Start with `artifacts/junit_operator.xml` to see which multi-stage test steps passed/failed — if the install step failed, the test never ran. Then check step logs under `steps//build-log.txt` and gathered resources under `gather/`. See [prow-reference.md](prow-reference.md) for the complete artifact layout, step descriptions, gather file listing, failure pattern table, and a mapping of Prow artifacts to their GitHub Actions equivalents. #### 4a. "No PipelineRun found" failures — PaC webhook chain When the failure is **"no pipelinerun found for component …"**, read [pac-webhook-debugging.md](pac-webhook-debugging.md) for detailed guidance on tracing the PaC webhook delivery chain. ### Step 5: Inspect the Tested Commit (Optional) This step is not always needed. Use it when the logs from Steps 3–4 suggest a regression in this repo's code (operator changes, kustomization updates, test modifications, deployment scripts) and you want to confirm by looking at the actual diff. Skip it when the root cause is already clear from the logs alone (e.g., an obvious infrastructure issue or upstream service crash). #### 5a. Get the head commit and base branch from the run **GitHub Actions:** ```bash # Show the head SHA, branch, and associated PR for the run gh run view --repo konflux-ci/konflux-ci --json headSha,headBranch \ --jq '"branch: \(.headBranch)\ncommit: \(.headSha)"' # Find the PR's base branch (don't assume main) gh pr list --repo konflux-ci/konflux-ci --head --json number,baseRefName \ --jq '.[0] | "PR #\(.number) → base: \(.baseRefName)"' ``` **Prow:** The commit info is in `started.json` and `prowjob.json` (downloaded in Step 2): ```bash # From started.json — shows the PR refs cat /tmp/prow-debug-/started.json | python3 -m json.tool # From prowjob.json — contains full refs spec cat /tmp/prow-debug-/prowjob.json | \ python3 -c "import json,sys; d=json.load(sys.stdin); r=d['spec'].get('refs') or d['spec']['extra_refs'][0]; print(f\"org: {r['org']}\nrepo: {r['repo']}\nbase: {r['base_ref']}\nSHA: {r['base_sha']}\nPR pulls: {r.get('pulls', [])}\")" ``` For presubmit (PR) jobs, the `pulls` array contains the PR number and head SHA. For periodic jobs, `base_sha` is the commit tested from the base branch. #### 5b. Check out and inspect the diff ```bash # Fetch the branch and check it out git fetch origin git checkout # View the full diff against the base branch git diff origin/... --stat git diff origin/... # If the PR has multiple commits, review them individually git log --oneline origin/.. ``` Scan the diff for changes to files related to the failure (e.g., kustomizations, operator code, test helpers, deployment scripts). Cross-reference with the error signals from Steps 3–4. #### 5c. Investigate git history for suspected regressions If a specific file or component looks suspicious, use git history to understand recent changes: ```bash # Recent commits touching a suspect file git log --oneline -20 -- # Show each change with diff git log -p -5 -- # Compare the suspect file between two points git diff -- ``` This is especially useful when the failure is new and the logs point to a particular controller, kustomization, or test file — tracing the history can pinpoint exactly which change introduced the regression. ### Step 6: Determine Root Cause and Suggest Fix Classify the failure into one of these categories: #### A. Test code issue (fix in this repo) The test assertion is wrong, flaky, or the test setup is incomplete. - Fix location: `test/go-tests/tests/conformance/` - Also check: `test/e2e/run-e2e.sh`, `deploy-test-resources.sh` #### B. Operator issue (fix in this repo) The Konflux operator itself is broken — reconciliation failures, bad defaults, missing RBAC. - Fix location: `operator/` (controllers, API types, RBAC, kustomize overlays) #### C. Upstream dependency issue (fix in upstream repo) A Konflux microservice is broken. Identify which service from the logs, then map it to the upstream repo: | Service | Upstream Repo | Kustomization | |---------|---------------|---------------| | Build Service | `konflux-ci/build-service` | `operator/upstream-kustomizations/build-service/` | | Integration Service | `konflux-ci/integration-service` | `operator/upstream-kustomizations/integration/` | | Release Service | `konflux-ci/release-service` | `operator/upstream-kustomizations/release/` | | Image Controller | `konflux-ci/image-controller` | `operator/upstream-kustomizations/image-controller/` | | Application API (CRDs) | `redhat-appstudio/application-api` | `operator/upstream-kustomizations/application-api/` | | Enterprise Contract | `conforma/crds` | `operator/upstream-kustomizations/enterprise-contract/` | Check the pinned commit SHA in the kustomization to identify which upstream version is deployed, then inspect that repo for relevant issues or recent changes. #### D. Infrastructure / flaky failure Resource exhaustion, network issues, Kind cluster instability. - Check `system-resources.log` and `cluster-resources.log` - If the test passed on one arch but failed on another, likely infra-related - If the failure is intermittent across runs, likely flaky #### E. Cross-job interference (shared external state) When a failure looks flaky, check whether **parallel CI jobs are competing over shared external resources**. The AMD64 and ARM64 e2e jobs run on separate clusters but may share the same external services (Quay organization, GitHub repos, OCI registries). Multiple PRs or retries can also run concurrently. If two jobs create, modify, or depend on the **same external resource** (e.g., same Quay repository path, same OCI artifact tag, same GitHub branch), one job can silently corrupt the other's state — overwriting images, rotating credentials, deleting data, or pushing conflicting artifacts. **How to check:** 1. Compare the `BeforeAll` setup logs from both arches (in their JUnit XML `` blocks) — look for any external resource identifiers (image URLs, repo paths, branch names, artifact names) that are identical. 2. Check if the failure is non-deterministic: one arch passes while the other fails, or results flip between re-runs with no code change. 3. Check whether other PRs or workflow runs were active at the same time and could have touched the same external resources. **Typical symptoms:** intermittent failures, "unauthorized" registry errors, wrong image content, unexpected data in pipeline results, one arch passes while the other fails, or tests that break only when CI is busy. **Fix:** ensure every concurrent job uses unique external resource names. See `operator/hack/setup-release.sh` `--image-name-prefix` (`-I`) for an example. ### Output Format Present findings as: ```markdown ## E2E Failure Analysis **Run:** **Failed job:** **Failed test(s):** ### Root Cause ### Evidence ### Category ### Suggested Fix - File(s) to change: ... - If upstream: repo + what to fix ``` ## Go toolchain failures If logs contain `go.mod requires go >=` or `running go 1.X` with a lower toolchain, switch to [go-toolchain-upgrade](../go-toolchain-upgrade/SKILL.md) — the fix is usually bumping Prow `build_root` / runner images in openshift/release or rebuilding `e2e-test-runner`, not only changing this repo. --- > Source: [konflux-ci/konflux-ci](https://github.com/konflux-ci/konflux-ci) — distributed by [TomeVault](https://tomevault.io).