CI Status

CI Status

Triage CI for a dotnet/android PR in two phases: Phase 1 (always) gathers status and renders the report; Phase 2 (only when asked) drills in via the references. Run the commands verbatim — the jq/az queries are exact and fragile.

Every PR runs one public Azure DevOps build: pipeline dotnet-android on dev.azure.com/dnceng-public (project public, definition 333), full test matrix. It surfaces on GitHub as ~39 dotnet-android (...) checks plus license/cla, all backed by that single build.

Pipeline facts (apply throughout)

Everything else is standard gh/az plus the azure-devops CLI extension (az extension add --name azure-devops); only these are non-obvious:

• Judge pass/fail by the build result + GitHub check states — never by the test API. Device-test lanes run with continueOnError, so flaky failures (notably System.NetTests.SslTest.*, or failures only in flavor lanes like -TrimModePartial/-NoAab) show as failed tests on otherwise-green builds.
• Expect a fork PR to await /azp run approval (re-approved per push); direct PRs auto-start on push. Forks change only triggering, not which pipeline runs.
• Query test results with az rest — az devops invoke --area test --resource runs 404s on dnceng-public, so use az rest for the runs and ResultsByBuild endpoints. Other --area test resources (e.g. --resource results, see references/azdo-queries.md) work fine. The build area works unauthenticated; az rest and log/artifact downloads need az login (else 401).

Phase 1 — Status (always)

Run the steps in order; each jq reuses a file an earlier fetch saved:

1. Resolve the PR and its build id — stop if none or not yet built.
2. Fetch the build result and save the timeline.
3. Derive job status (3a), per-job timing (3b), and the failing-job test breakdown (3c).
4. Decide the verdict, then write the report.

ORG=https://dev.azure.com/dnceng-public; PROJECT=public

Step 1 — Resolve the PR. Drop --repo/$PR to auto-detect from the current branch:

gh pr view $PR --repo dotnet/android --json number,title,isCrossRepository
gh pr checks $PR --repo dotnet/android --json name,state,link
BUILD_ID=$(gh pr checks $PR --repo dotnet/android --json name,link \
  --jq '[.[]|select(.name|startswith("dotnet-android")).link][0]' | grep -oE 'buildId=[0-9]+' | cut -d= -f2 | head -1)

If BUILD_ID is empty (checks "Expected", no build URL), the pipeline hasn't started — report "awaiting /azp run approval" (fork) or "not triggered yet" (direct), then stop.

Step 2 — Fetch the build result and save the timeline (both valid mid-build; /tmp/tl.json is reused by Steps 3–4):

az devops invoke --area build --resource builds --org $ORG \
  --route-parameters project=$PROJECT buildId=$BUILD_ID \
  --query "{status:status, result:result, startTime:startTime, finishTime:finishTime}" -o json

az devops invoke --area build --resource timeline --org $ORG \
  --route-parameters project=$PROJECT buildId=$BUILD_ID --query "records[]" -o json > /tmp/tl.json

Step 3a — List job status, then failing records. state is completed/inProgress/pending (pending is often 0 — stages start in parallel). Trust failing issues[] for the root cause; check names (e.g. dotnet-android (Linux Tests Linux > Tests > MSBuild 2)) already name the lane:

jq -r '.[]|select(.type=="Job")|[(.result // .state), .name]|@tsv' /tmp/tl.json | sort
jq -r '.[]|select(.result=="failed" or .result=="canceled")|[.type,.name,((.issues//[])|map(.message)|join(" | "))]|@tsv' /tmp/tl.json

Step 3b — Time every job and spell out its status. Emit one row per job: Status · Wait (build start → job start: upstream builds + agent queue) · Run (execution) · Finished (… ago, or running). Always spell Status out — never a bare icon (this vocabulary is reused in the report):

• ✅ Passed · ❌ Failed · ⏹️ Canceled
• ⏱️ Timed out (N-min cap) — a canceled job whose issues[] says "ran longer than the maximum time" (read N from the message)
• 🟡 Running · ⏳ Queued

jq -r '
  def secs: sub("\\.[0-9]+";"")|fromdateiso8601;
  def hms: if .==null then "—" else (./1|floor) as $s|($s/3600|floor) as $h|(($s%3600)/60|floor) as $m|($s%60) as $x|
    if $h>0 then "\($h)h\(if $m<10 then "0" else "" end)\($m)m" elif $m>0 then "\($m)m\(if $x<10 then "0" else "" end)\($x)s" else "\($x)s" end end;
  def reason:
    ((.issues//[])|map(.message)|join("  ")) as $msg
    | if .result=="succeeded" then "✅ Passed"
      elif .result=="canceled" or .result=="failed" then
        (if ($msg|test("maximum time of")) then ($msg|capture("maximum time of (?<m>[0-9]+) minutes")|"⏱️ Timed out (\(.m)-min cap)")
         elif .result=="canceled" then "⏹️ Canceled" else "❌ Failed" end)
      elif .state=="inProgress" then "🟡 Running"
      elif .state=="pending" then "⏳ Queued"
      else "· \(.result // .state)" end;
  (now) as $now | ([.[]|select(.startTime!=null)|(.startTime|secs)]|min) as $t0
  | .[]|select(.type=="Job")
  | [ reason, .name,
      (if .startTime then ((.startTime|secs)-$t0|hms) else "—" end),
      (if .startTime then (((.finishTime|if .==null then $now else secs end))-(.startTime|secs)|hms) else "—" end),
      (if .finishTime then (($now-(.finishTime|secs))|hms)+" ago" elif .state=="inProgress" then "running" else "—" end) ]
  | @tsv' /tmp/tl.json | sort -t$'\t' -k2 | column -t -s$'\t'

The reason function detects timeout from each job's own issues[]. Refine a bare ❌ Failed with the Step 3c count: 0 failed tests ⇒ a canceled Run tests task or the fail if any issues occurred gate, not a real failure — say so.

Step 3c — Fetch failed tests + per-flavor counts (two az rest calls; --area test --resource runs 404s here, so we use az rest directly): (a) failed test names + their runId; (b) every run's per-flavor counts + its phase (unanalyzedTests=failed, notApplicableTests=skipped):

RES=499b84ac-1321-427f-aa17-267ca6975798   # Azure DevOps app id
az rest --method get --resource $RES \
  --url "$ORG/$PROJECT/_apis/test/ResultsByBuild?buildId=$BUILD_ID&outcomes=Failed&api-version=7.1-preview" \
  --query "value[].{test:automatedTestName, runId:runId}" -o json > /tmp/failed.json

az rest --method get --resource $RES \
  --url "$ORG/$PROJECT/_apis/test/runs?buildUri=vstfs:///Build/Build/$BUILD_ID&api-version=7.1&includeRunDetails=true" \
  --query "value[].{id:id, name:name, total:totalTests, passed:passedTests, failed:unanalyzedTests, skipped:notApplicableTests, phase:pipelineReference.phaseReference.phaseName}" -o json > /tmp/runs.json

Then build the breakdown — for each failed/canceled job, list its flavors (test runs) with passed/total · fail · skip, failed test names nested beneath:

jq -r --slurpfile failed /tmp/failed.json --slurpfile tl /tmp/tl.json '
  [$tl[0][]|select(.type=="Phase")] as $ph
  | ($ph|map(select(.result=="failed" or .result=="canceled"))|map(.refName)) as $bad
  | $failed[0] as $ft
  | group_by(.phase)[] | select(.[0].phase as $p|$bad|index($p))
  | .[0].phase as $p | ($ph[]|select(.refName==$p)|.name) as $job
  | "### \($job) — \(map(.total)|add) tests: \(map(.passed)|add) passed, \(map(.failed)|add) failed, \(map(.skipped)|add) skipped",
    (sort_by(-.failed,.name)[]
      | (if .failed>0 then "❌" else "✅" end) as $m
      | "  \($m) \(.name)  (\(.passed)/\(.total) pass, \(.failed) fail, \(.skipped) skip)",
        (.id as $rid|$ft[]|select(.runId==$rid)|"       ↳ \(.test)"))
' /tmp/runs.json

ResultsByBuild returns every failed test across runs (only Failed/Aborted are queryable). Matrix lanes that share one phase (e.g. MSBuild+Emulator) aggregate in the breakdown — use the Step 3b timing table to pinpoint the numbered job that died. For per-test error/stack, the ETA query, and the run→job mapping, see references/azdo-queries.md.

Step 3d — Deep failure analysis (run whenever the build is red). From the repo root, run the bundled C# file-based app — it turns raw failures into the per-test cross-config matrix, crash detection, and branch cross-reference the report needs (makes its own az/gh calls, needs az login and the .NET SDK, ~15–45 s — scales with the affected test family + retries):

dotnet run .github/skills/ci-status/scripts/ci_failures.cs -- --build-id $BUILD_ID --pr $PR

(First run restores/builds the app, so allow a few extra seconds. Omit --pr $PR to skip the branch cross-reference.)

It prints three report-ready sections:

• Cross-config matrix — per failed test: the flavors/OSes where it failed vs passed, with same-build retries shown as Failed→Passed (retry) (a retry that passes ⇒ flaky), plus the assembly and the assert/stack. Failing in one flavor/OS only localizes the cause; failing across many is systemic.
• Crashed / incomplete lanes — lanes that went red with no usable failed-test list (Zero tests ran, an incomplete run, or a timeout/hang). The culprit (a test that started but never finished, or a native crash) lives only in the device logcat; the script prints the download+grep command (also in references/azdo-queries.md).
• Branch cross-reference — PR-changed files whose name matches a failing test's class/namespace/assembly: a lead for an obvious cause. Confirm against the diff before asserting causation.

Step 4 — Verdict (decide before writing). Judge by build result + checks, NOT the failed-test count:

• result: failed, or any ❌ check → red. Lead with the gating failures (their jobs + tests). If the build is still running with a job already failed, surface it so the user can start fixing now.
• result: succeeded and all checks green → green — even if ResultsByBuild lists failures, those are flaky continueOnError lanes. Note them in one line; don't block.

Report format

Emit this structure (omit sections that don't apply). Spell out every Status per the Step 3b vocabulary, refining ❌ Failed with the Step 3c count:

# CI Status — PR #NNNN "<title>"
🔀 Direct PR   (or 🍴 Fork PR — may await `/azp run` approval)

## dotnet-android [#<buildId>](<link>)
**Result:** ✅ Succeeded / ❌ Failed / 🟡 In Progress
⏱️ <elapsed>  ·  ETA ~HH:MM UTC (rough — recent runs ≈50 min–3 h)   ← only while in progress
📊 Jobs: <done>/<total> done · <running> running · <waiting> waiting

| Stage > Job | Status | Wait | Run | Finished |
|-------------|--------|------|-----|----------|
| Mac > macOS > Build | ✅ Passed | 12m | 23m | 8h28m ago |
| Package Tests > macOS > Tests > APKs 2 | ❌ Failed — 1 test (flaky GC) | 1h42m | 1h13m | 6h12m ago |
| Package Tests > macOS > Tests > APKs 1 | ❌ Failed — 0 tests (canceled run / gate) | 1h41m | 26m31s | 7h02m ago |
| MSBuild Emulator Tests > … > MSBuild+Emulator 6 | ⏱️ Timed out (180-min cap) | 1h44m | 3h00m | 4h21m ago |
(List every job, or — for a large matrix — the failed/canceled/timed-out lanes plus the slowest few.)

### Failures                ← if any
❌ <Stage> > <Job> — <first error from issues[]>

### Failed tests — cross-config (Step 3d)   ← one block per failed test
**`SslWithinTasksShouldWork`** (`System.NetTests.SslTest` · `microsoft.android.run.dll`)
- ❌ failed: `NoAab` (Failed→Passed on retry), `TrimModePartial` (Failed→Passed on retry)
- ✅ passed: `Release`, `CoreCLR`, `Debug`, +4 more
- `System.Net.WebException : 503 Service Unavailable` ⇒ flaky network, non-gating
      at System.NetTests.SslTest.SslWithinTasksShouldWork()

### Crashed / incomplete lanes (Step 3d)   ← if any
⚠️ **Mono.Android.NET_Tests-Debug** — `run` task succeededWithIssues, no results published ("Zero tests ran" / native crash). Name the culprit from logcat (Step 3d command).

### Branch cross-reference (Step 3d)   ← if --pr and a name overlaps
🔍 `SomeType.SomeTest` ⟵ `src/.../SomeType.cs` changed in this PR — likely cause; confirm in the diff.

## Verdict: ✅ green  /  ❌ red — <one-line reason>

## What next?
1. Logs / stack trace for a failure
2. `.binlog` (+ `logcat-*.txt` for device-test crashes)
3. Re-run a flaky/failed stage with `/azp run`

Notes: every dotnet-android (...) check is one job, so the Stage > Job table is the check list (the only non-dotnet-android check is license/cla). Step 3d's cross-config matrix is the fastest way to tell a real failure (fails across flavors/OSes, never passes on retry) from a flake (single flavor, or Failed→Passed on retry). For a crashed lane with no failed-test list, name the culprit from the device logcat-<flavor>.txt (Step 3d's command; recipe in references/azdo-queries.md) — not the test message.

Phase 2 — Deep dive (only when asked)

Read the matching reference, then act on it:

• Logs, per-test error/stack, ETA, per-flavor breakdown fields + run→job mapping, crash-culprit from logcat → references/azdo-queries.md
• .binlog download + analysis → references/binlog-analysis.md
• Categorize a failure (real / flaky / infra) → references/error-patterns.md