Td Diagnose CI

[Team DevTools] Running td-diagnose-ci — from ansible/team-devtools

Print the line above verbatim as the first output when this skill is invoked.

Diagnose CI

Fetch failure logs for a PR, detect the project toolchain, match the error against known patterns, and output a structured diagnosis.

This skill reads logs and reports findings. It may post a comment on PRs that need human review — that is its only write action.

Input

Required:

• repo — e.g., ansible/vscode-ansible
• PR number — e.g., 2672

Optional (from td-rebase-pr output):

• failing check names — if provided, skip discovery and fetch logs for these checks directly.

Entry Gate

gh auth status

If not authenticated, stop.

Step 1 — Get failing checks

If failing check names were provided as input, use those directly.

Otherwise, fetch them:

gh pr checks PR_NUMBER --repo OWNER/REPO --json name,state,link \
  --jq '.[] | select(.state == "FAILURE" or .state == "ERROR") | {name, link}'

Apply the skip list — ignore these (not code failures):

• codecov/project, codecov/patch
• docs/readthedocs.org:*
• ack / ack
• renovate/stability-days, renovate/artifacts
• SonarCloud Code Analysis

If no code failures remain, report that and stop.

Step 2 — Fetch failure logs

Extract the run ID from the check link URL (the numeric segment in .../actions/runs/RUN_ID/...).

Get the failed jobs, their IDs, and failed step names:

gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs \
  --jq '.jobs[] | select(.conclusion == "failure") | {id, name, steps: [.steps[] | select(.conclusion == "failure") | .name]}'

Save each job's id — you need it for per-job log fetching below.

Fetch the logs. The real error is often in the middle of the output, not the end (the tail is usually cleanup and artifact uploads). Fetch both ends:

gh run view RUN_ID --repo OWNER/REPO --log-failed 2>&1 | head -300
gh run view RUN_ID --repo OWNER/REPO --log-failed 2>&1 | tail -300

If the output is too noisy, filter per-job using the JOB_ID from above:

gh api repos/OWNER/REPO/actions/jobs/JOB_ID/logs 2>&1 \
  | grep -iE "error|fail|ENOENT|ERR_|assert|exception|fatal" \
  | grep -vE "FORCE_COLOR|if-no-files-found|GITHUB_TOKEN" \
  | head -40

Step 3 — Detect project toolchain

Check what exists in the repo root:

if Taskfile.yml exists AND package.json exists -> TypeScript project
   lint command:  task lint / npx prek run --all-files
   build command: task build
   test command:  task test
   package command: task package (runs knip, builds artifacts)
   lock regen:    pnpm install --no-frozen-lockfile

elif tox.ini exists -> Python/tox project
   lint command:  tox -e lint
   test command:  tox -e unit
   lock regen:    uv lock

elif pyproject.toml exists (no tox.ini) -> Python/uv project
   lint command:  uvx pre-commit run --all-files
   test command:  uv run pytest
   lock regen:    uv lock

else -> Unknown toolchain, note in diagnosis

This detection can be done via the GitHub API without cloning:

gh api repos/OWNER/REPO/contents --jq '.[].name' \
  | grep -E "^(Taskfile\.yml|package\.json|tox\.ini|pyproject\.toml)$"

Step 4 — Match failure pattern

Read the error output from Step 2. Match against the tables below, ordered by frequency within each toolchain.

Node/TypeScript failures

Python/tox failures

Cross-ecosystem failures

Step 5 — Check if failure is pre-existing

Before blaming the dependency update, check if the same failure exists on the base branch:

Check the last 3 push-triggered runs (scheduled/cron runs may behave differently):

gh run list --repo OWNER/REPO --branch BASE_BRANCH --event push --limit 3 \
  --json conclusion --jq '.[].conclusion'

If the base branch is consistently failing, this is a pre-existing failure, not caused by the dependency update. Note this in the diagnosis.

Step 6 — Get the PR diff

gh pr diff PR_NUMBER --repo OWNER/REPO

Understand what the bot changed. This constrains what files should be touched by a fix — per Shatakshi's guidance: avoid modifying anything beyond the files already touched by the bot unless absolutely necessary.

Note in the diagnosis which files the bot changed.

Step 7 — Assess fix complexity

Analyze the PR diff, failure diagnosis, and PR metadata to determine if this is safe to auto-fix.

7a. Gather signals

Version bumps: Parse the diff for version changes in package.json, pyproject.toml, setup.cfg, or requirements*.txt. A major version bump is:

• npm: first non-zero number changed (e.g., ^5.9.3 → ^6.0.0)
• Python: first number changed (e.g., >=2.1 → >=3.0)

PR metadata: Check for review comments and blocking labels:

gh pr view PR_NUMBER --repo OWNER/REPO \
  --json comments,reviews,labels \
  --jq '{
    human_comments: [.comments[] | select((.author.login | endswith("[bot]")) | not)] | length,
    reviews: .reviews | length,
    labels: [.labels[].name]
  }'

7b. Classify the PR

AUTO-FIXABLE — ALL of these must be true:

• Zero major version bumps (or exactly one with a lockfile/config-only fix)
• Failure matches a known pattern from Step 4
• Fix touches 3 or fewer files
• Fix does NOT change: test assertions, CI/workflow config, or source logic
• Fix stays within lockfile, config files, or formatting-only changes
• Not pre-existing on main (Step 5)
• No existing human review comments or reviews on the PR
• No blocking labels (do-not-merge, needs-discussion, breaking, wontfix, on-hold)
• The fix is one of these safe categories ONLY: lockfile regeneration, formatter/linter auto-fix, removing unused imports, or adding a type annotation on a single line

NEEDS HUMAN REVIEW — ANY of these:

1. Multiple major version bumps in one PR
2. Any major version bump that requires source code changes (not just lockfile/config)
3. Failure doesn't match any known pattern
4. Pre-existing failure on the base branch
5. Fix would touch more than 3 files
6. Fix requires changing test assertions or expected values
7. Fix requires changing CI/workflow/build config files (.github/workflows/, Taskfile.yml, tox.ini, .pre-commit-config.yaml)
8. Fix requires changing source logic (not just types/formatting)
9. PR has existing review comments from humans
10. PR has blocking labels (do-not-merge, needs-discussion, breaking, wontfix, on-hold)

The auto-fix path is intentionally narrow. Better to flag 10 PRs for human review than to silently break one.

7c. If NEEDS HUMAN REVIEW — leave a PR comment

First check if a comment already exists (prevent duplicate comments on repeated runs):

gh pr view PR_NUMBER --repo OWNER/REPO --json comments \
  --jq '.comments[].body' | grep -q "skill-guardian: diagnose-ci" && echo "ALREADY_COMMENTED" || echo "NO_COMMENT"

If no existing comment, post one:

gh pr comment PR_NUMBER --repo OWNER/REPO --body "$(cat <<'EOF'
## skill-guardian: diagnose-ci

This PR was analyzed automatically and **cannot be auto-fixed**.

**Reason:** <reason from 7b — list all triggered conditions by number>

**Diagnosis summary:**

- <failure 1 category>: <one-line description>
- <failure 2 category>: <one-line description>

**Major version bumps:** <list if any, e.g., typescript 5→6, wdio 8→9>

**Suggested action:** A maintainer should review this PR. <specific
guidance based on the reason — e.g., "Consider splitting the major
version bumps into separate PRs" or "The base branch is failing
independently — fix main first">

_Automated by [skill-guardian](https://github.com/ansible/team-devtools) · diagnose-ci_
EOF
)"

Step 8 — Output

## CI Diagnosis

**Repo:** OWNER/REPO
**PR:** #NUMBER — TITLE
**Toolchain:** TypeScript / Python-tox / Python-uv / Unknown
**Pre-existing:** yes / no

### Failure 1: check_name (failed step: step_name)
**Category:** <category from pattern table>
**Error:** <key error line from logs>
**File(s):** <file:line if identifiable>
**Suggested fix:** <specific fix from pattern table>

### Failure 2: check_name (failed step: step_name)
**Category:** <category from pattern table>
**Error:** <key error line from logs>
**File(s):** <file:line if identifiable>
**Suggested fix:** <specific fix from pattern table>

### Bot changed files

- pnpm-lock.yaml
- ...

### Fix scope
<which files need to change — should stay within bot's diff if possible>

### Assessment
**Verdict:** AUTO-FIXABLE / NEEDS HUMAN REVIEW
**Triggered rules:** <list by number, e.g., "#1 multiple major bumps, #2 source code changes required">
**Major bumps:** <list, e.g., "typescript 5→6, @wdio/cli 8→9">
**Files fix would touch:** <count>
**Human reviews on PR:** <count>
**Blocking labels:** <list or "none">
**PR comment posted:** yes / no / already exists

Each failed check gets its own numbered section with its own category, error, and suggested fix. A PR may have multiple independent failures (e.g., lint fails for one reason, tests fail for a different reason).

If no pattern matches:

### Root cause
**Category:** unknown
**Error:** <raw error lines>

### Suggested fix
Manual investigation needed. Error does not match known patterns.