# Nightly Analyze

> Nightly Failure Analysis

- Skill: `qualcomm/nightly-analyze` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qualcomm/nightly-analyze`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qualcomm/nightly-analyze/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qualcomm (https://skillmd.com/u/qualcomm)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/qualcomm/nightly-analyze

---

# Nightly Failure Analysis

Analyze nightly CI failures and post a structured comment on the GitHub issue.

**Budget: You have ~35 tool calls. Be efficient. Batch queries. Do NOT read files one-by-one.**

## Context Variables

Injected via the workflow prompt:
- `RUN_ID` — the failed workflow run ID
- `REPO` — the repository (owner/name)
- `RUN_URL` — direct link to the workflow run

## Critical `gh` CLI Rules

- ALWAYS use `--json` and `--jq` flags — never parse human-readable text
- NEVER use `!=` in `--jq` expressions — bash mangles `!`. Use `select(.conclusion == "failure")` instead
- Do NOT use `gh run view --log-failed <PARENT_RUN_ID>`. Breeze runs *inside*
  the workflow it analyzes, so the parent run is always "in progress" and the
  command returns `run <id> is still in progress; logs will be available when
  it is complete`. Use the per-job endpoint from Step 1.5 instead — job logs
  are available the moment the job finishes.

## Sandbox Gotchas

These are real constraints in the Breeze runner — ignoring them wastes turns on
permission denials and command-parsing failures.

- **Writable paths**: redirect to flat files under `/tmp/` (e.g.
  `/tmp/breeze-<job_id>.log`), NOT into a subdirectory like `/tmp/claude/`.
  The runner's write allowlist is `/__w/<repo>/<repo>` and `/tmp` at file
  granularity — `> /tmp/claude/foo.log` is blocked even though
  `mkdir -p /tmp/claude` used to be granted.
- **No pipes in Bash**: `cmd | head -10` triggers a permission denial because the matcher
  splits on `|` and rechecks each side. Use the tool's own flags instead (`--limit`,
  `--jq '[.[]] | .[0:10]'`, `head -n 10 file.txt` against a saved file).
- **No newline-chained commands**: `cmd > file\necho "exit: $?"` counts as
  command chaining and is denied like `;` or `&&`. Run each command as its
  own Bash call.
- **Quote `gh api` URLs that contain `&`**: bash parses unquoted `&` as a background
  operator and splits the command. Always:
  ```
  gh api "repos/owner/name/commits?path=X&per_page=10" --jq '...'
  ```
  Not:
  ```
  gh api repos/owner/name/commits?path=X&per_page=10 --jq '...'   # DENIED
  ```

## Step 1: Get Failed Jobs and Test Results (~3 tool calls)

1. Get failed jobs in ONE call:
   ```
   gh run view $RUN_ID --json jobs --jq '.jobs[] | select(.conclusion == "failure") | {name, conclusion}'
   ```

2. Try downloading test results artifact:
   ```
   gh run download $RUN_ID -n nightly-test-results -D /tmp/nightly-results
   ```

3. If download succeeds, read `/tmp/nightly-results/summary.md` — it has pre-built failure tables with test names, errors, and stack traces. This is your primary data source. If download fails, or the summary.md does not name a specific failing test/error for one of the failing jobs, proceed to Step 1.5 before categorizing.

## Step 1.5: When JUnit is empty, READ THE ACTUAL LOG (mandatory)

If a failing job has 0 JUnit failures, or summary.md does not name a
specific failing test/error, you MUST fetch that job's raw log before
proposing any root cause. Do not skip this and guess from commit history.

For each failing job, use its `databaseId` from Step 1 and run the
`fetch_job_log.py` helper — one Bash call per job, no pipes, no
compound shell:

    python3 scripts/breeze_nightly/fetch_job_log.py <JOB_ID> --tail 50

The helper writes the raw log to `/tmp/breeze-<JOB_ID>.log`, all matches
to `/tmp/breeze-<JOB_ID>-hits.txt`, and prints the last 50 filtered
hits to stdout so you can consume them directly. Only Read the raw log
file when you need more context around a specific hit.

The log names the failing model, the missing binary/module, and the
exact call site. That is your ground truth. Only after you have a
concrete error string from the log may you proceed to Step 3 (commit
bisection) and Step 4 (routing).

## Step 2: Summarize + Categorize Failures (~3 tool calls)

From summary.md or job data, group failures:

| Category | Detection |
|----------|-----------|
| Unit Test | `*-unit-tests-junit.xml` or "QAIHM Tests" |
| Model Test | `*-model-tests-junit.xml` or "Model Tests" — nightly runs each model in a fresh per-model venv, so `ModuleNotFoundError`, `ImportError`, or an `environment_setup` failure in the JUnit XML almost always means a manifest bug (missing dep, broken `pre_pip_install_commands`), NOT a model bug. Route to the model's owner. |
| Workbench Job | `*-verify-workbench-jobs-junit.xml` |
| Workflow/Infra | Jobs that failed without XML |
| Cross-Version | Same test fails across 3.10/3.11/3.12/3.13 |

**Dedup rule:** Same test failing across all Python versions = report once as "cross-version".

For each category: count, unique error signatures, first stack trace (max 10 lines).

## Step 3: Find Breaking Commits (~5 tool calls)

1. Find last successful nightly + current SHA in ONE call each:
   ```
   gh run list --workflow=nightly.yml --status=success --limit=1 --json headSha,createdAt,databaseId
   gh run view $RUN_ID --json headSha --jq '.headSha'
   ```

2. Get commit range with file changes in ONE call:
   ```
   gh api repos/$REPO/compare/{last_sha}...{current_sha} \
     --jq '.commits[] | {sha: .sha[0:8], date: .commit.author.date[0:16], author: .commit.author.name, message: .commit.message | split("\n")[0], files: [.files[].filename]}'
   ```
   **If this doesn't return files**, get the stat summary instead:
   ```
   gh api repos/$REPO/compare/{last_sha}...{current_sha} --jq '{total_commits: .total_commits, files: [.files[].filename]}'
   ```

3. Cross-reference changed files with failing tests. Do NOT check each commit individually.
   For each suspect commit, note which specific failure(s) it relates to (use the failure # from the summary table).
   **PR references:** Always use fully-qualified format `qcom-ai-hub/ai-hub-models-internal#N` — comments are posted on tetracode issues, so bare `#N` resolves to the wrong repo.
   General rules:
   - `models/<id>/*` changed → `models/<id>/test*` failures
   - `utils/`, `configs/`, `test/` changed → unit test failures
   - `global_requirements.txt`, `pyproject.toml` changed → cross-version failures
   - `scorecard/` changed → scorecard failures

4. If 0 commits between last success and current failure, note: "No new commits. Likely external dependency, flaky test, or infrastructure issue."

5. **Collapse-when-no-match rule (output noise control):** if NO commit in the range plausibly touches the failing files (i.e. every row would be marked "Unrelated"), do NOT emit the Suspect Commits table. Replace it with a single line, e.g. `**Suspect Commits:** N commits in range; none touch failing files (last passing \`sha\` → current \`sha\`).` Only emit the full table when ≥1 commit is plausibly related to a failure.

## Step 4: Root Cause Analysis + Triage (~5 tool calls)

**STOP-AND-POST RULE — non-negotiable:**
If you have used ~30 tool calls and have not yet started Step 5 (posting the comment),
STOP investigating immediately. Post the comment with whatever you have, marking
unresolved failures as confidence `LOW` or `MEDIUM`. A best-effort analysis posted on
the issue is far more useful to the czar than a perfect analysis that hits the turn cap
and posts nothing. The goal is to deliver, not to be exhaustive.

**First, check `.claude/triage/historical-patterns.md` for known recurring patterns.**
Many nightly failures match historical signatures (transient host outages, service timeouts,
dependency breakages). If the failure matches a known pattern, classify it immediately
without deeper investigation. Only dig into commits/code if the pattern is novel.

**CRITICAL — Find the root cause BEFORE proposing any fix.**

Do NOT propose workarounds that mask the real issue. A fix that tolerates bad data is a
bandaid — the right fix addresses WHY the data is bad in the first place.

NEVER propose a root cause naming a specific dep/model/PR unless you have
quoted an exact error string from a log fetched via Step 1.5. "Zero JUnit
failures suggests collection abort" is a *category*, not a root cause — the
log names the actual failing import. If every Step 1.5 fetch failed, post
with confidence `LOW` and say "unable to read logs from sandbox; root cause
not identified" — do not guess from commit history alone.

**Root cause checklist:**
1. **Where does the unexpected data/state originate?** Trace the error upstream:
   - If our code crashes on unexpected input, ask: is the input wrong, or is our code wrong?
   - If a dependency produces unexpected output (e.g. renamed tensors, wrong formats),
     the fix belongs in that dependency — file a ticket to the responsible team.
   - If our code is too strict/too loose for valid behavior, the fix is in our code.
2. **Search for existing issues BEFORE proposing a fix:**
   ```
   gh issue list --repo qcom-ai-hub/tetracode --search "<model_name> OR <error_keyword>" --state open --limit 5 --json number,title,labels,assignees
   ```
   If an existing issue tracks this failure, reference it instead of proposing a new fix.
3. **Is this a bug in an external dependency?** (AIMET, QNN compiler, Hub API, etc.)
   - If yes: recommend filing a ticket to the owning team with the job ID and repro steps.
   - Only recommend a workaround in our code if clearly labeled as temporary.
   - **AIMET signals:** `QcQuantizeOp_` prefix, `_q` suffix, `w8a8`/`w8a16` precision failures
     → Route to `Quantization` team. Do NOT propose fuzzy-match fixes in our code.
4. **Is this a regression from a recent PR?** Cross-reference with suspect commits from Step 3.

**In your output, always state:**
- **Root cause:** one sentence on what's actually wrong and where
- **Owner:** which team/component owns the fix
- **Recommended action:** file ticket / fix in our code / both (temporary workaround + ticket)

Use `.claude/triage/` files for routing. Key decision process:

**Check error origin FIRST:**

1. **Stack trace in `qai_hub_models/`** → likely `ai-hub-models`, BUT check what produced the bad state:
   - `QcQuantizeOp_` prefix in tensor names → `Quantization` (AIMET bug, not ours)
   - Compiler renamed outputs (no `QcQuantizeOp_`) → `Compiler/ONNX2EP`
   - Our logic error on valid data → `ai-hub-models`

2. **External system error** → route per `.claude/triage/error-patterns.md`:
   - Compile failures ("Cannot capture", shape errors) → `Compiler/ONNX2EP`
   - Context binary exit codes (malformed binary) → `Compiler/ONNX2EP`
   - QNN runtime crash ("NPU crashed", "graph execute error") → `Tungsten`
   - TFLite delegate issues → `Compiler/ONNX2EP`
   - OOM / timeout / HTTP 5xx from Hub → `Cloud services`

3. **Dependency / transient** → see `.claude/triage/historical-patterns.md`

**Error severity:**
- ImportError/ModuleNotFoundError → **Blocking** (dependency change)
- TypeError/AttributeError → **Blocking** (API change)
- TimeoutError/connection errors → **Non-blocking** (transient, re-run)
- OOM/exit 137 → **Blocking** (`Cloud services`)

**NEVER assign to a specific person — the czar rotates weekly.**

## Step 5: Emit Comment for Downstream Posting (~2 tool calls)

You do NOT post the comment directly. The Breeze runner's `GITHUB_TOKEN` is
repo-scoped to `ai-hub-models-internal` and cannot write to
`qcom-ai-hub/tetracode` — `gh issue comment` will 404 and burn turns. Instead,
a separate `post_breeze_comment` job (with `STAGING_GH_TOKEN`) recovers your
comment from this job's log and posts it to every URL in `ISSUE_URLS`.

Your only post step from here is a two-command emit:

1. Write the full comment markdown (format below) to a flat `/tmp/` file:
   ```
   Write /tmp/breeze_comment.md
   ```
   NOT `/tmp/claude/...` — subdirectory writes under `/tmp/` are blocked by the
   sandbox. Flat filename only.

2. Emit the file base64-encoded between recovery markers (ONE Bash call):
   ```
   python3 scripts/breeze_nightly/emit_comment_b64.py /tmp/breeze_comment.md
   ```

That's the entire post-step from your side. Do NOT call `gh issue comment` or
`gh api /repos/qcom-ai-hub/tetracode/...` — you don't have the credentials, and
retries will exhaust the turn cap. Keep the comment under 65,000 characters.

## Output Format

```markdown
## Breeze AI Nightly Analysis

**Run:** [View Workflow]($RUN_URL) | **Date:** YYYY-MM-DD | **Failures:** X across Y suites

---

### Failure Summary

| Category | Count | Key Error | Python Versions |
|----------|------:|-----------|-----------------|
| ... | ... | ... | ... |

<details>
<summary>Category Name (N failures)</summary>

| Test | Error | File |
|------|-------|------|
| ... | ... | ... |

</details>

---

### Suspect Commits

**Last passing:** YYYY-MM-DD HH:MM — `sha` | **Current failing:** YYYY-MM-DD HH:MM — `sha` | **Commits in range:** N

| Commit | Date (UTC) | Author | Message | Related Failure | Suspect? |
|--------|------------|--------|---------|-----------------|----------|
| ... | MM-DD HH:MM | ... | ... | #1 (model_name) | reason |

---

### Root Cause Analysis & Triage

| # | Failure | Root Cause | Owner | Recommended Action | Severity |
|---|---------|------------|-------|-------------------|----------|
| ... | ... | ... | ... | ... | ... |

> Soft recommendations for the nightly czar. Do not assign to individuals.
> If the root cause is in an external dependency, file a ticket to the owning team.

---
**Grading this triage:** when you close this issue, please apply one of
`triage-correct` / `triage-wrong` / `triage-transient` so the weekly KB-update
agent can score itself against human ground truth. See
[LABELING.md](https://github.com/qcom-ai-hub/ai-hub-models-internal/blob/main/.claude/triage/LABELING.md).

*Generated by Breeze AI nightly analyst*
```

**Optional Reasoning Trace — include ONLY when warranted.**
Most nightly failures match a known pattern with HIGH confidence. In those cases, do NOT
add a reasoning trace — it just spells out harness steps and adds noise for the czar.

Append the block below ONLY when at least one of the following is true:
- The dominant failure has confidence < HIGH (genuine ambiguity remains).
- The failure does not match any entry in `historical-patterns.md` (novel signature).
- Team routing was non-obvious (e.g. ruled out 2+ teams before settling).
- The Suspect Commits table was emitted (i.e. ≥1 commit flagged as Suspect).

When you do include it, keep it terse — list only what is true for THIS failure. Skip
sub-bullets that have nothing to say.

```markdown
<details>
<summary>Agent Reasoning Trace</summary>

**Patterns matched:** which `error-patterns.md` / `historical-patterns.md` entries matched, with confidence (HIGH/MEDIUM/LOW).
**Team routing:** why this team; what was ruled out (only if non-obvious).
**Commit bisection:** which files changed correlated with which tests (only if a commit was flagged Suspect).
**Uncertainties:** any failures where confidence < HIGH.

**Job logs:** [Full agent output]($RUN_URL) (see "AI Nightly Failure Analysis" job)

</details>
```

## Rules

- Batch `gh` calls — never loop over commits one-by-one
- If JUnit XML parsing fails, fall back to summary.md; if that fails, use job-level data
- Be concise — the czar needs actionable info, not prose
- Include UTC timestamps
- Keep comment under 65,000 characters
- Use fully-qualified cross-repo references (`qcom-ai-hub/ai-hub-models-internal#N`) for all PR/issue numbers — comments are posted on `qcom-ai-hub/tetracode`, so bare `#N` links to the wrong repo