Triage CI Failure

Triage SPDK CI Failure

Use this skill when a Gerrit patch has a CI failure that needs to be triaged. Given a Gerrit change number (e.g. 28816), the skill walks through:

1. Fetching the change details and finding the failed build
2. Downloading and analyzing the failed job logs
3. Searching GitHub issues for a matching known failure
4. Drafting a new issue if no match exists
5. Printing the Gerrit comment to post

Prerequisites

• gh CLI authenticated to GitHub (gh auth login) with access to spdk/spdk-ci (private) and spdk/spdk (public)
• curl and jq available on the system
• Internet access to review.spdk.io and github.com

Step 1: Fetch Gerrit Change Details

Query the Gerrit REST API (no auth required for public changes) and extract the CI build messages.

CHANGE=<patch_number>

# Fetch change detail (response is prefixed with )]}' which must be stripped)
curl -s "https://review.spdk.io/changes/${CHANGE}/detail" \
  | tail -c +5 \
  | jq '.' > /tmp/gerrit_change_${CHANGE}.json

Parse the messages array to find the latest message from SPDK Automated Test System that contains Build failed:

jq -r '
  .messages[]
  | select(.author.username == "spdk-bot")
  | select(.message | test("Build failed"))
  | {date: .date, message: .message, revision: ._revision_number}
' /tmp/gerrit_change_${CHANGE}.json

From the message, extract the run ID and attempt number. The format is:

Build failed. Results: [RUN_ID/ATTEMPT](URL)

Extract with:

# Get the latest failed build message
FAILED_MSG=$(jq -r '
  [.messages[]
   | select(.author.username == "spdk-bot")
   | select(.message | test("Build failed"))]
  | last
  | .message' /tmp/gerrit_change_${CHANGE}.json)

RUN_ID=$(echo "$FAILED_MSG" | grep -oP 'actions/runs/\K[0-9]+')
ATTEMPT=$(echo "$FAILED_MSG" | grep -oP 'attempts/\K[0-9]+')
PATCHSET=$(jq -r '
  [.messages[]
   | select(.author.username == "spdk-bot")
   | select(.message | test("Build failed"))]
  | last
  | ._revision_number' /tmp/gerrit_change_${CHANGE}.json)

echo "Run ID: $RUN_ID, Attempt: $ATTEMPT, Patchset: $PATCHSET"

If there is no Build failed message, stop and tell the user -- there is nothing to triage.

Step 2: Identify Failed Jobs

List all jobs for the run and find which ones failed:

gh run view "$RUN_ID" --attempt "$ATTEMPT" -R spdk/spdk-ci

This shows a summary of all jobs with their status. Note the job name(s) that show as X (failed). Common job names include:

Note: Job summary / merge_outputs and Job summary / report failures are usually secondary -- they fail because an upstream job failed. Focus on the primary failed job (the test job that actually ran tests).

Step 3: Download Failed Job Logs

Download the full logs zip via the GitHub API (the gh run view --log-failed flag is unreliable and often returns empty output):

gh api "repos/spdk/spdk-ci/actions/runs/${RUN_ID}/attempts/${ATTEMPT}/logs" \
  > /tmp/ci_logs_${RUN_ID}.zip

List the contents to find the failed job's log file:

unzip -l /tmp/ci_logs_${RUN_ID}.zip

Log files are named like 4_HPE NVMe-oF tests _ hpe-nvmf-rdma.txt or 2_Nvidia NVMe-oF tests _ nvidia-nvmf-rdma.txt. Find the one matching the failed job from Step 2.

Important: The tail of the log contains the actual failure. Early lines contain expected test errors (e.g. memory map tests, lock contention tests) that are normal. Always start from the end:

# Get the last 500 lines -- the real failure is at the end
unzip -p /tmp/ci_logs_${RUN_ID}.zip "*FAILED_JOB_PATTERN*" | tail -500

# Then narrow down to error lines in the tail
unzip -p /tmp/ci_logs_${RUN_ID}.zip "*FAILED_JOB_PATTERN*" | tail -500 \
  | grep -iE 'ERROR|FAILED|exit code|backtrace|assert|timeout|SIGTERM|SIGKILL|aborting'

Focus on:

1. The backtrace section near the end -- shows the exact failure path
2. *ERROR* lines in the last ~200 lines -- the real error, not test noise
3. Failing Code: block -- shows the exact line that triggered the trap
4. exit 1 / exit code lines -- shows propagation path

Step 4: Analyze the Failure

From the logs, identify:

1. Failed test name -- the specific test or sub-test that failed, e.g. nvmf_rdma.nvmf_target_core_interrupt_mode.nvmf_ns_hotplug_stress
2. Error signature -- the key error message, e.g. nvme_qpair.c:722 nvme_qpair_abort_queued_reqs *ERROR*
3. Failure context -- what was happening when the failure occurred
4. Job/runner name -- e.g. hpe-nvmf-rdma, nvidia-nvmf-rdma, nvidia-nvmf-tcp, autorun-bdev-vm

Build a set of search keywords from this analysis. Good keywords include:

• The failed test function/script name
• The error message (file:line, function name)
• The failed job name
• Key error strings (timeout, assertion, signal name)

Step 5: Search GitHub Issues

Search for matching issues in spdk/spdk. Start with the Intermittent Failure label, then broaden if nothing matches.

5a. Search by label + keywords

# Search with Intermittent Failure label and keywords
gh issue list -R spdk/spdk \
  -l "Intermittent Failure" \
  --search "<keyword from error>" \
  --state open \
  --limit 20

Try multiple keyword combinations:

# By job name
gh issue list -R spdk/spdk \
  -l "Intermittent Failure" \
  --search "hpe-nvmf-rdma" --state open
# or for NVIDIA jobs
gh issue list -R spdk/spdk \
  -l "Intermittent Failure" \
  --search "nvidia-nvmf-rdma" --state open
# By test name
gh issue list -R spdk/spdk \
  -l "Intermittent Failure" \
  --search "nvmf_ns_hotplug_stress" --state open
# By error function
gh issue list -R spdk/spdk \
  -l "Intermittent Failure" \
  --search "nvme_qpair_abort_queued_reqs" --state open

5b. Broaden search if no label match

# Search all open issues (not just Intermittent Failure)
gh issue list -R spdk/spdk --search "<keyword>" --state open --limit 20
# Also check closed issues (may have been fixed but regressed)
gh issue list -R spdk/spdk --search "<keyword>" --state closed --limit 10

5c. Read candidate issues

For each candidate issue, read its body to verify the failure matches:

gh issue view <issue_number> -R spdk/spdk --json title,body,labels

Compare:

• Is the failed job name the same?
• Is the error message / function the same or similar?
• Is the test name the same?
• Does the failure pattern match (same backtrace, same exit path)?

An issue matches if the error signature is substantially the same, even if the patch under test is different (intermittent failures are by definition patch-independent).

Step 6: Report the Result

6a. If a matching issue was found

Print the Gerrit comment for the user to post on the patch:

false positive: ISSUE_NUMBER

For example: false positive: 3902

Also explain briefly why this failure matches the known issue so the user can verify the match.

6b. If no matching issue was found

Draft a new GitHub issue following the existing format. Present the draft to the user for approval before creating.

Title format: [test_name] Brief error description

Examples of good titles (from existing issues):

• [fio_dif_rand_params] heap-use-after-free in uring_sock_group_impl_poll
• [nvmf_target_disconnect_tc2] Unable to reset the controller
• [build failure] Unable to acquire the dpkg frontend lock
• HPE NVMe-oF tests / hpe-nvmf-rdma (job-level failure)
• Nvidia NVMe-oF tests / nvidia-nvmf-rdma (job-level failure)

Body format (following existing convention from issues like #3902):

# CI Intermittent Failure

<Job name> -> <test path that failed>.

<Brief description of the failure: what happened, where in the code, and the
immediate trigger.>

## Link to the failed CI build

[(<change>/<patchset>)<subject> - spdk/spdk-ci@<sha>](<github_actions_url>)

## Execution failed at

<dotted test path, e.g. nvmf_rdma.nvmf_target_core_interrupt_mode.nvmf_ns_hotplug_stress>

<relevant error log excerpt -- keep it focused, 20-50 lines max>

Label: Intermittent Failure

Show the draft to the user and ask for confirmation. Once approved, create:

gh issue create -R spdk/spdk \
  --title "<title>" \
  --body "<body>" \
  --label "Intermittent Failure"

After creation, print the Gerrit comment:

false positive: NEW_ISSUE_NUMBER

Quick Reference

Triage Workflow:
  1. Fetch Gerrit change  -- curl review.spdk.io/changes/{N}/detail
  2. Extract failed build -- parse spdk-bot "Build failed" message
  3. Identify failed job  -- gh run view {RUN_ID} --attempt {ATTEMPT}
  4. Download logs        -- gh api .../logs > zip; unzip | tail
  5. Analyze failure      -- error signature, test name, context
  6. Search GitHub issues -- gh issue list -l "Intermittent Failure"
  7. Match or draft issue -- compare signatures; draft if new
  8. Print Gerrit comment -- "false positive: ISSUE_NUMBER"

Common Failure Patterns

Example Gerrit Comments

# Simple reference to known issue
false positive: 3902

# With brief context (optional)
false positive: 3902
hpe-nvmf-rdma nvmf_ns_hotplug_stress -- qpair abort

# NVIDIA job example
false positive: 4100
nvidia-nvmf-rdma nvmf_target_disconnect_tc2 -- controller reset