Triage Integration Test Failures

Triage integration test failures

Overview

A failure investigation ends with one of five remedies, not a patch guessed under time pressure. Classify the failure first. The category decides the remedy; the wrong category produces a fix that treats a symptom.

REQUIRED BACKGROUND: superpowers:systematic-debugging governs root-cause work in general. This skill adds the classification step and the toolkit specific to this repo's CI, agent, and test infrastructure.

Hard rule: delegate the reading

A CI job log, an az monitor/kubectl dump, and an agent log are evidence, not the deliverable. None of them belongs in the main session.

• Run every command in the toolkit below inside a subagent (Agent tool -- general-purpose or Explore), not in the main session. Give the subagent the exact command and the exact question ("did this job fail on a connection error before any agent code ran, or on an assertion diff?"); it returns a short verdict with the one or two evidence lines that prove it, never the raw log or the full command output.
• This applies even to a "quick look" -- a CI log or agent log runs to hundreds of KB or has single lines tens of KB wide, and once that text lands in the main session it is re-billed on every later turn for the rest of the investigation.
• For agent-log evidence specifically, dispatch a subagent that follows the analyze-dotnet-agent-logs skill; do not hand-parse the log yourself even inside that subagent.
• The main session's job is Steps 2-4: collect verdicts, classify, decide, and record the decision. It holds conclusions, not evidence bytes.

Step 1: is it even a flake?

• Same PR, same test, one run failed and a rerun passed with no code change: flake. Continue.
• Fails on every run, or only after a real code change: this is a regression, not a flake. Use superpowers:systematic-debugging instead.
• Check gh run list for the workflow: has this exact test or job failed before on unrelated PRs? Count occurrences of the exact signature across recent scheduled runs, and state the rate as per-run, not per-call -- one occurrence in 100 runs is not proof of absence. A recurring signature across unrelated changes is strong evidence of an environmental cause, not a code bug.

Step 2: classify by evidence, not guess

Before matching a row, trace the failing message or exception string to the code that emits it, and note whether the assertion sits in the test body or in fixture setup/teardown (e.g. RemoteApplicationFixture.TestForKnownProblems is a health check, not the test's own logic) -- this decides which row applies.

Never conclude "just flaky" without pulling at least one row's evidence. See [[feedback_no_guessing]] -- a root cause claim needs a log line, a metric, or a matching upstream issue behind it, not a hunch. An assertion that exists to catch a product defect is never reclassified as timing/race and its threshold is never widened -- trace and fix the defect instead.

Step 3: pick the remedy for the category

Step 4: record the decision

Every one of the categories above ends in a decision someone will ask about again ("why don't we just fix the LB exposure", "did coverlet ever get bumped"). Capture, in a project memory or handoff note:

1. The failure signature and the run/job that showed it.
2. The evidence that ruled out the other categories.
3. The remedy chosen, and what was explicitly rejected and why.
4. Anything left open (a version to watch for, a follow-up not started).

Toolkit quick reference

Each row runs inside a subagent per the hard rule above; the main session sees only the verdict it returns.

• Both gh run view --job <id> --log-failed and --log truncate silently past ~889 lines / 168KB, cutting off mid agent-log dump. For the complete text: gh api repos/<org>/<repo>/actions/jobs/<jobid>/logs. Copy any log to a file before a step that clears it -- lost evidence cannot be re-created. The pristine agent log also survives as CI artifact integration-test-results-<matrix> (all_solutions.yml uploads C:\IntegrationTestWorkingDirectory\**\*.log unconditionally).
• az monitor metrics list --resource <aks-or-lb-id> --metric <name> -- cluster/LB health at the failure timestamp (MSYS_NO_PATHCONV=1 needed for the slash-prefixed resource ID in Git Bash).
• kubectl get pods -n unbounded-services -o wide -- pod restarts/uptime after az aks get-credentials.
• analyze-dotnet-agent-logs skill -- agent-side evidence (harvest timing, Seen/Sent counts, connect sequence) from the test's own log.
• run-integration-tests skill -- reproduce the failing test locally before trusting a fix. A null result only counts if the code path is proven to have executed (log entry/exit counts); when the agent itself is a suspect, rerun once with the agent detached to isolate the confound. Delegate the run itself, per that skill's own subagent guidance; take back the summary, not the console output.

Common mistakes

• Treating "all variants failed the same way" as proof of a code bug -- it is usually the opposite: identical failure across unrelated variants points at a shared external dependency, not the code under test.
• This pattern covers whole-class failures across unrelated variants only -- a single test failing once, or a fixture health-check assertion firing, is not covered by it and may be the product-defect row instead.
• Fixing the assertion's timeout/retry count instead of the mechanism that makes the wait racy. If a similar bump has already happened once on this test, that is the signal to stop bumping and find the mechanism.
• Closing the investigation without writing down why the infra issue was not hardened -- the same question comes back next time the flake recurs.
• Reading a CI log or agent log directly in the main session "just this once" -- delegate it every time; a single raw read is how a multi-step investigation quietly fills the context window.