# Fix CI

> Check CI status, analyze test failures, auto-fix obvious issues or discuss with user

- Skill: `questdb/fix-ci` (Agent Skill)
- Install (CLI): `npx skillmds@latest add questdb/fix-ci`
- Raw SKILL.md: https://api.skillmd.com/api/skills/questdb/fix-ci/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: questdb (https://skillmd.com/u/questdb)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/questdb/fix-ci

---


Analyze CI failures for PR `$ARGUMENTS` (or the current branch's PR if no argument given).

## Step 1: Discover build status

### 1a. Find the PR

```bash
# If $ARGUMENTS is a number or URL, use it directly.
# Otherwise detect from current branch:
gh pr view --json number,title,headRefName --jq '{number, title, headRefName}'
```

### 1b. Quick test summary (cheapest check, 1 API call)

Parse the buildId from any check URL first (see Step 1c), then hit the test summary endpoint:

```bash
curl -s "https://dev.azure.com/{org}/{project_guid}/_apis/test/resultsummarybybuild?buildId={buildId}&api-version=7.0-preview"
```

No auth needed. Returns aggregate counts instantly (~1KB):
```json
{
  "aggregatedResultsAnalysis": {
    "totalTests": 279518,
    "resultsByOutcome": {
      "Passed": {"count": 276837},
      "Failed": {"count": 24},
      "NotExecuted": {"count": 2657}
    },
    "runSummaryByOutcome": {
      "Failed": {"runsCount": 14},
      "Passed": {"runsCount": 23}
    }
  }
}
```

If `Failed.count == 0` and all checks passed — report success, stop.

### 1c. Fetch all CI checks

Use `gh pr checks` **text format** (tab-separated) — it reliably includes Azure URLs unlike the JSON `statusCheckRollup` where `detailsUrl` can be null.

```bash
gh pr checks $PR 2>&1
```

Output format (tab-separated):
```
CheckName\tstatus\tduration\tURL
```

Parse and classify each check:
- `pass` — succeeded
- `fail` — failed
- `pending` — still running

Group checks by source:
- **Azure Pipelines** — URL contains `dev.azure.com`. Extract buildId from first Azure URL. These are the test jobs.
- **GitHub Actions** — URL contains `github.com/actions`. These include `build` (compilation), `gitleaks` (secret scanning), `Danger` (PR linting). Failed GitHub Actions checks are relevant — report them.
- **Other** — `CodeRabbit` (no URL or review-only) — skip.

Extract buildId from the first Azure URL (all checks in one pipeline run share the same buildId).

### 1d. Report non-Azure failures

For each failed GitHub Actions check, fetch its details:

```bash
# Extract run_id and job_id from URL: https://github.com/{owner}/{repo}/actions/runs/{run_id}/job/{job_id}
gh run view {run_id} --json jobs --jq '.jobs[] | select(.conclusion == "failure") | {name, conclusion}'
```

Report these as Category E (non-test failures) in the final output. Common cases:
- **Danger** — PR convention issues (title format, description, labels). Show the comment: `gh pr view $PR --comments --jq '.comments[-1].body'`
- **build** — compilation failure. The job log contains the error.
- **gitleaks** — the job failed on a credential-shaped string in one of the
  branch's commits, or on a scan error; tell the two apart before reporting.
  A `leaks found: N` line in the job log means the scan ran and found N; a clean
  scan writes `no leaks found`, with no colon, so the colon form is the signal.
  The Actions API exposes no job summary, but this repo's job tees ready-to-paste
  ignore lines to the log, so fetch them with
  `gh run view {run_id} --log | cut -f3- | cut -d' ' -f2- | grep -E '^[^ ]+:[a-z0-9-]+:[0-9]+$' | sort -u`.
  Anchor on that line shape, not a `grep -A N` window around the heading: the
  runner echoes the workflow's own `run:` body into the log, so a heading match
  returns script source first, and a fixed window silently truncates a long
  finding list. Only this repo emits those lines: the enterprise and client
  repos run the same action with no reporting step, so read their findings from
  gitleaks' own verbose block, which every repo prints, with
  `gh run view {run_id} --log | cut -f3- | cut -d' ' -f2- | grep -E '^(RuleID|File|Line):'`.
  Nothing back from either, and no `leaks found:` line, means the job failed
  before the scan reported anything; read the log before naming why.
  - `fatal: Invalid revision range` or `fatal: ambiguous argument ... unknown
    revision`, alongside `ERROR: Unexpected exit code [1]`, is gitleaks failing
    on a commit range the checkout cannot resolve. The action pins the workspace
    to the SHA in the event payload, then asks the API for the pull request's
    commits seconds later, so anything moving the branch head in between hands
    it a ref the checkout does not carry — an ordinary push (the range head is
    merely ahead) or a force-push (the histories diverge). Either way gitleaks
    adds `partial scan completed` and `no leaks found in partial scan` over a
    scan of nothing, so that pair does not tell the two apart. The push that
    caused it starts a run of its own, so the next run clears it and there is
    nothing to fix.
  - `FTL unable to load gitleaks config` (exit 1), or a Go `panic:` trace from
    `regexp: Compile` (exit 2), means `.gitleaks.toml` itself is broken — an
    unbalanced `(` in a `[[rules.allowlists]]` regex is enough. The panic aborts
    gitleaks before it writes `results.sarif`, and the action then dies on the
    missing file while uploading it as an artifact, so this job produces no
    findings table, no job summary at all, and no `Leaks detected` warning — the
    log is the only record. Name the config error and do not expect a re-run to
    clear it.
  - `::error::.gitleaks.toml detected nothing` (this repo only) is the canary
    step rather than the scan. The scan above ran and reported cleanly, but a
    throwaway token the default rules must catch went undetected, so
    `.gitleaks.toml` is loading no rules and the branch's commits were never
    really read. There is no finding, no `leaks found:` line and no fingerprint
    to fetch. Name the config error; a re-run will not clear it either.

  Report the raw error in all three cases; do not call it a leaked credential.
  With findings, report them and do not act on them: like every Category E
  failure, the decision is the user's (Step 5). A real credential has to be
  rotated first, and only a confirmed false positive gets suppressed by
  committing those `<file>:<rule-id>:<start-line>` lines to `.gitleaksignore`.
  The scan runs with `--redact`, so neither the log nor the SARIF carries the
  value and you cannot make that call yourself. Never propose the four-part
  `Fingerprint:` line from the log — it names a commit the squash-merge
  discards, so it silences the PR and then fails the push scan of `master`.

### 1e. Triage

- **All passed + summary shows 0 failures** — report success, stop.
- **Some failed** — proceed to Step 2 with failed checks. Report summary: "24 test failures across 14 runs out of 279K tests."
- **All pending, none failed** — report "CI still running, no failures yet. Re-run this command later or use `/loop 2m /fix-ci $PR` to monitor."
- **Mix of pending + failed** — proceed with failed checks immediately. Don't wait for the full run.
- **Only non-Azure checks failed** (e.g., Danger, build) — report those directly, skip Steps 2-3.

## Step 2: Get failed test names (no auth, no log download)

Use the Azure Test Results microservice at `vstmr.dev.azure.com`. This is a different hostname from the build APIs and serves test result data publicly for public projects.

```bash
curl -s "https://vstmr.dev.azure.com/questdb/questdb/_apis/testresults/resultsbybuild?buildId={buildId}&publishContext=CI&outcomes=Failed&\$top=200&api-version=5.2-preview.1"
```

**No authentication or special headers needed.** Returns all failed test results (~1-5KB):

```json
[
  {
    "automatedTestName": "test[/sql/sample_by_fill.test]",
    "automatedTestStorage": "io.questdb.test.sqllogictest.SqlTest",
    "outcome": "Failed",
    "runId": 795795,
    "durationInMs": 1067.0,
    "id": 100001,
    "testCaseTitle": "test[/sql/sample_by_fill.test]"
  }
]
```

Key fields:
- `automatedTestName` / `testCaseTitle` — the test name
- `automatedTestStorage` — the test class (e.g., `io.questdb.test.sqllogictest.SqlTest`)
- `runId` — which CI job run this failure came from
- `outcome` — always "Failed" given the filter

**Deduplicate by test name** — the same test fails across multiple platforms (mac, windows, linux). Group by `automatedTestName`, collect `runId`s to know which platforms failed.

If this returns 0 results but Step 1b showed failures, fall back to Step 2b (log tail parsing). This can happen if the pipeline doesn't publish JUnit test results.

### 2a (optional). Enrich with error messages via PAT

The `vstmr` endpoint returns test names but NOT `errorMessage`, `stackTrace`, or `computerName`. If `AZURE_DEVOPS_PAT` is set, enrich each failed test with full details.

Step 2 gives us `runId` per failed test. Use the authenticated `test/runs/{runId}/results` endpoint to get error details:

```bash
# For each unique runId from Step 2:
curl -s -u ":$AZURE_DEVOPS_PAT" \
  -H "X-TFS-FedAuthRedirect: Suppress" \
  "https://dev.azure.com/questdb/questdb/_apis/test/runs/{runId}/results?outcomes=Failed&api-version=7.0"
```

This returns full details per failed test:
- `errorMessage` — the assertion failure or exception message
- `stackTrace` — full Java stack trace
- `automatedTestName`, `automatedTestStorage` — test identity
- `failingSince` — when this test started failing
- `failureType` — type of failure

With this data, skip directly to Step 4 (classification).

Note: `test/runs?buildId=...` (listing runs by build) requires `Build: Read` scope and returns 403 with `Test Management: Read` alone. But `test/runs/{runId}/results` works with just `Test Management: Read` — and we already have runIds from the unauthenticated Step 2.

**If no PAT is set**, suggest the user create one for richer failure data:

> To get error messages and stack traces without downloading logs, set `AZURE_DEVOPS_PAT`:
>
> 1. Go to https://dev.azure.com/questdb/_usersSettings/tokens
> 2. Click "New Token"
> 3. Set scope: **Test Management → Read** (the only scope needed)
> 4. Add to `~/.zshenv`: `export AZURE_DEVOPS_PAT=<token>`
>
> Without it, I can still see which tests failed but need to download log tails for error details.

Only suggest this once per session, and only if log tail parsing is actually needed (i.e., the test names alone aren't enough for classification).

### 2b. Fallback: log tail parsing (no auth)

Use this when Step 2 returns 0 results or when error messages are needed and no PAT is available.

#### Parse Azure URLs

Extract org, project GUID, buildId, jobId from each failed check's URL:
```
https://dev.azure.com/{org}/{project_guid}/_build/results?buildId={buildId}&view=logs&jobId={jobId}
```

All failed checks share the same buildId. Deduplicate: fetch the timeline only once per buildId.

### 2c. Fetch timeline

```bash
curl -s "https://dev.azure.com/{org}/{project_guid}/_apis/build/builds/{buildId}/timeline?api-version=7.0"
```

No authentication needed (public project).

Parse the JSON response. Records form a tree: Stage -> Job -> Task.
For each failed check's jobId:
1. Find all records where `type == "Task"` AND `parentId == jobId` AND `result == "failed"`
2. Extract `name` and `log.id` from each failed task

### 2d. Classify failed steps

- **"Run tests"** or **"Run tests with Coverage"** — test failure, proceed to log analysis
- **"Compile with Maven"** — compilation error, report to user directly
- Other steps — infrastructure failure (checkout, install, upload), report as-is

### 2e. Get log line counts

```bash
curl -s "https://dev.azure.com/{org}/{project_guid}/_apis/build/builds/{buildId}/logs?api-version=7.0"
```

Response: `{"value": [{"id": N, "lineCount": M, ...}, ...]}`. Extract lineCount for each failed step's logId.

## Step 3: Fetch and parse error summaries

### 3a. Download tail of each failed test log

Maven Surefire writes the error summary at the END of its output. Use the line-range API to fetch only the last ~500 lines:

```bash
curl -s "https://dev.azure.com/{org}/{project_guid}/_apis/build/builds/{buildId}/logs/{logId}?api-version=7.0&startLine={lineCount - 500}&endLine={lineCount}"
```

This fetches ~50KB instead of ~200MB. Save to a temp file for parsing.

### 3b. Parse the Surefire summary

Look for these patterns in the downloaded tail:

**Test error summary** (exceptions during test execution):
```
[ERROR] Errors:
[ERROR]   ClassName.testMethod:lineNum->...chain... >> ExceptionType message
```

**Test failure summary** (assertion mismatches):
```
[ERROR] Failures:
[ERROR]   ClassName.testMethod:lineNum expected:<X> but was:<Y>
```

**Totals line**:
```
[ERROR] Tests run: N, Failures: M, Errors: K, Skipped: L
```

**Compilation error** (different pattern entirely):
```
[ERROR] COMPILATION ERROR
[ERROR] /path/to/File.java:[line,col] error: ...
```

For each failed test, extract:
- Fully-qualified class name (e.g., `SqlParserTest`, `WindowFunctionTest`)
- Test method name
- Error type: assertion mismatch vs exception
- Error message / exception chain
- Line number in test source (from the `:lineNum` in the chain)

### 3c. Deduplicate across jobs

The same test might fail on multiple platforms (mac-griffin, windows-griffin, linux-griffin). Group failures by `ClassName.testMethod` — if the error message is the same across platforms, it's one logical failure. Note which platforms are affected.

## Step 4: Classify failures

### 4a. Get PR diff

```bash
gh pr diff $PR
```

Parse to understand:
- Which source files changed (production code vs test code)
- What functions/methods were modified
- What behavior changes the PR introduces

### 4b. Cross-reference each failure group with the PR diff

For each failed test group, determine the category:

**Category A — Auto-fixable** (ALL must hold):
1. The failure is an assertion mismatch (`Failures:` section, not `Errors:` section)
2. The PR modifies production code in the area the test covers (same package, same class, related function)
3. The PR already updated similar test assertions in the same or other test files (pattern exists in the diff)
4. The fix is mechanical: swap the expected value to match actual output
5. **Guard-removal gate**: if a test changed from "expected to fail" to "now succeeds" (e.g., error test that no longer errors), check the PR diff for removed guards (methods like `guardAgainst...`, `throw SqlException` blocks, early-return checks). If the PR removed a guard WITHOUT replacing the underlying functionality, the test likely exposes an unhandled code path — escalate to Category B/D. Only auto-fix if the PR replaced the guarded code with a new implementation that handles the case.

The heuristic: "Did the PR replace the functionality, or just remove the gate?" Replacing → auto-fix. Removing gate without replacement → discuss.

**Category B — Behavior precision**:
- The failure connects to PR changes (related classes/packages)
- But the PR did NOT already update similar tests, so intent is unclear
- Or the failure is an exception (not just a different value) that might indicate the test expected the old behavior to continue
- Or a guard was removed and the test now succeeds where it previously failed — need to verify the old code path is actually handled

**Category C — Potential regression**:
- The failing test is in a package/class NOT touched by the PR
- No clear connection between the test's subject and the PR's changes

**Category D — Potential incompleteness**:
- The test covers an edge case (NULL input, empty table, boundary value, special characters)
- The PR introduced new logic but the test suggests it doesn't handle this case
- Often: test expected a query to succeed, but new code throws an exception for this input

**Category E — Non-test failure**:
- Compilation error
- Infrastructure issue (timeout, OOM, disk full, network error)
- Flaky test (known flaky pattern, random ordering issue)

## Step 5: Act

### For Category A (auto-fix):

1. **Find the test source file:**
   - Convert class name to path: `io.questdb.test.griffin.FooTest` -> search in `core/src/test/java/`
   - Use Glob: `**/FooTest.java`

2. **Get the expected vs actual values:**
   - If the Surefire summary contains the full assertion diff (common with `assertEquals`): use directly
   - If the summary only has an exception message (common with `assertQueryNoLeakCheck`): need to search the full log
   - To search: use line-range chunks. First, find the `<<< FAILURE!` line for this test method:
     ```bash
     # Search in chunks of 50K lines from the end, looking for the test method name + FAILURE
     curl -s "...logs/{logId}?startLine={lineCount-50000}&endLine={lineCount}" -o /tmp/ci-chunk.txt
     grep -n "testMethodName" /tmp/ci-chunk.txt | grep -i "FAILURE\|ERROR\|expected\|but had"
     ```
   - Read +-100 lines around the match to get the full assertion diff

3. **Read the test method** in the source file. Find the assertion call and its expected value.

4. **Update the expected value** to match the actual output. Use the Edit tool.

5. **Report** what was changed:
   ```
   Auto-fixed: FooTest#testBar
   - Updated expected output: [brief description of what changed]
   - Reason: PR changed [behavior X], test expected old output
   - Platforms affected: mac-griffin, windows-griffin, linux-griffin
   ```

### For Categories B-E (discuss with user):

Present a structured report. Group by category, within each category group by similarity.

```markdown
## CI Failures: PR #NNN — [PR title]

Analyzed N failed jobs across M platforms.

### Auto-fixed (if any)
- `FooTest#testBar`: updated expected output — [description]

### Needs Discussion

#### Potential Regression (Category C)
Tests in areas NOT touched by this PR:
- `BarTest#testQux`: NullPointerException at SomeClass.java:42
  Platforms: linux-griffin, mac-griffin
  [Stack trace summary]

#### Potential Incompleteness (Category D)
New logic may not handle these cases:
- `WindowFunctionTest#testWindowAsArg`: SqlException "Window function is not allowed in context of aggregation"
  The PR added [feature X] but these tests show queries that combine window functions with aggregation.
  Platforms: all

#### Behavior Precision (Category B)
Connected to PR changes but need review:
- `SqlParserTest#testWindowFuncOrder`: expected query model differs from actual
  The PR changed [parser behavior X]; this test may need updating or may reveal unintended side effect.

#### Non-test Failures (Category E)
- Job `windows-cairo-2`: "Compile with Maven" step failed — compilation error in FooBar.java:123
```

After presenting, ask the user how to proceed with each group.

## Azure API Reference

### Test Results (vstmr.dev.azure.com) — no auth, no special headers

The test results microservice lives on a separate hostname. No authentication or special headers needed.

| Endpoint | Returns |
|---|---|
| `.../resultsbybuild?buildId={id}&publishContext=CI&outcomes=Failed&$top=200&api-version=5.2-preview.1` | Array of failed test results: `automatedTestName`, `automatedTestStorage`, `outcome`, `runId`, `durationInMs` |
| `.../resultdetailsbybuild?buildId={id}&publishContext=CI&groupBy=TestRun&$filter=Outcome eq Failed&shouldIncludeResults=true&queryRunSummaryForInProgress=false&api-version=5.2-preview.1` | Failed results grouped by test run, with counts per outcome |

Base: `https://vstmr.dev.azure.com/questdb/questdb/_apis/testresults`

### Test Result Details (dev.azure.com) — PAT with `Test Management: Read`

| Endpoint | Returns |
|---|---|
| `https://dev.azure.com/questdb/questdb/_apis/test/runs/{runId}/results?outcomes=Failed&api-version=7.0` | Full details: `automatedTestName`, `errorMessage`, `stackTrace`, `failingSince`, `failureType` |

Auth: `curl -u ":$AZURE_DEVOPS_PAT"`. Note: `test/runs?buildId=...` (listing runs) needs `Build: Read` scope, but `test/runs/{runId}/results` works with just `Test Management: Read` since we get runIds from the unauthenticated vstmr call.

### Test Summary (dev.azure.com) — no auth

| Endpoint | Returns |
|---|---|
| `https://dev.azure.com/questdb/{project_guid}/_apis/test/resultsummarybybuild?buildId={id}&api-version=7.0-preview` | Aggregate counts: total, passed, failed, not executed |

### Build APIs (dev.azure.com) — no auth

Base: `https://dev.azure.com/questdb/{project_guid}/_apis/build`

The project GUID is embedded in check URLs. Parse it from there rather than hardcoding.

| Endpoint | Returns |
|---|---|
| `/builds/{buildId}/timeline?api-version=7.0` | `{records: [{id, parentId, type, name, result, state, log: {id}, order}]}` |
| `/builds/{buildId}/logs?api-version=7.0` | `{value: [{id, lineCount, createdOn}]}` |
| `/builds/{buildId}/logs/{logId}?api-version=7.0&startLine=N&endLine=M` | Plain text, lines N through M |

### Timeline record types
- `Stage` — pipeline stage (parent of Jobs)
- `Job` — a CI job (parent of Tasks), maps to a GitHub check
- `Task` — a step within a job, has `log.id` for log download

### Status mapping
- `result="succeeded"` → success
- `result="failed"` → failure
- `result="skipped"` → skipped
- `result="canceled"` or `"cancelled"` → cancelled
- `state="completed"` with no result → success
- Otherwise → pending