Kb Weekly Update

Weekly Knowledge Base Update

Review resolved nightly and scorecard issues from the past week, evaluate triage accuracy for both, and propose updates to the triage knowledge base via a PR.

Budget: 50 tool calls max. Target: finish in under 40 tool calls. Batch aggressively.

Context Variables

Injected via the workflow:

• REPO — the repository (owner/name), e.g. qcom-ai-hub/ai-hub-models-internal
• TETRACODE_REPO — issue tracker, e.g. qcom-ai-hub/tetracode
• LOOKBACK_DAYS — how many days to look back (default: 7)

Step 1: Collect Data (~12–18 tool calls)

For ~7 nightly issues, the per-issue body+comments+timeline trio costs ~21 calls. Batch all per-issue calls into a single message so the runtime parallelizes them. If approaching the 45-action wrap-up threshold, finish evaluation with whatever evidence you have rather than fetching more.

Run these FOUR calls in the SAME message (parallel):

Call A — nightly issues from the past week:

gh issue list --repo $TETRACODE_REPO \
  --search "[QAIHM Nightly] created:>=$(date -d '$LOOKBACK_DAYS days ago' +%Y-%m-%d)" \
  --label "ai-hub-models" \
  --state all --limit 20 \
  --json number,title,state,createdAt,closedAt,labels

Call B — scorecard regression issues from the past week (and one extra week back, so we can check whether last week's SUSTAINED predictions persisted):

gh issue list --repo $TETRACODE_REPO \
  --search "label:scorecard created:>=$(date -d '$((LOOKBACK_DAYS + 7)) days ago' +%Y-%m-%d)" \
  --state all --limit 20 \
  --json number,title,state,createdAt,closedAt,labels

Call C — merged PRs from the past week:

gh pr list --repo $REPO \
  --search "is:merged merged:>=$(date -d '$LOOKBACK_DAYS days ago' +%Y-%m-%d)" \
  --limit 50 \
  --json number,title,author,mergedAt,body,files

Call D — read all 5 KB files in one message:

cat .claude/triage/error-patterns.md .claude/triage/teams.md \
    .claude/triage/runtime-guide.md .claude/triage/examples.md \
    .claude/triage/scorecard-patterns.md

Then, for each issue (nightly + scorecard), fetch THREE things in parallel — body, comments, and timeline — so verdicts rest on positive evidence, not on silent closure. Batch all issues into one message:

# Per issue (parallel; replace {num} with each issue number):
gh api repos/$TETRACODE_REPO/issues/{num} --jq '{number, state, title, body, labels: [.labels[].name], assignee: .assignee.login, closed_at}'
gh api repos/$TETRACODE_REPO/issues/{num}/comments --jq '[.[] | {author: .user.login, created: .created_at, body: .body[0:8000]}]'
gh api repos/$TETRACODE_REPO/issues/{num}/timeline --paginate --jq '[.[] | select(.event=="closed" or .event=="assigned" or .event=="referenced" or .event=="cross-referenced") | {event, actor: .actor.login, assignee: .assignee.login, source: .source.issue.number, commit_id: .commit_id, created: .created_at}]'

From the body, extract the failing tests / job IDs / stack traces — this is the ground-truth signal the nightly Breeze was triaging from. You must re-read it; do not take the nightly Breeze's RCA on faith.

From the comments, classify each one:

• Nightly agent comments — contain "Breeze AI Nightly Analysis" or "Generated by Breeze AI"
• Scorecard agent comments — contain "Breeze AI Scorecard Analysis"
• Human follow-ups — anything from a *_QCOM author. Read these in full. They are the only source of authoritative ground truth.

From the timeline, look for these decisive signals:

• referenced / cross-referenced events → fixing PRs that reference this issue
• A closed event whose commit_id is set → closed by a commit (likely a fix)
• Re-assigns (assigned events with different assignee) → the original triage was rejected by the first assignee; the routing pattern in the agent's RCA is suspect
• A closed event with no commit_id, no preceding fix-PR reference, and no human comment → silent closure. This is the most common pattern AND the most over-credited. Default such issues to UNVERIFIED in Step 2a, NEVER VERIFIED-CORRECT.

For nightly issues, final labels + assignee are weak signals (they may just mean nobody re-routed it). The strong signals are: a fix PR whose author matches the agent's named team, OR a human comment confirming / refuting the agent's RCA.

Step 2: Evaluate + Discover (~8 tool calls)

2a. Nightly accuracy

For each nightly issue that had an agent triage comment, your job is to verify the agent's RCA against positive evidence. A silent closure is NOT confirmation — it is absence of signal.

Required cross-check (do this for every issue, no exceptions):

0. Check the issue's labels first (highest-priority signal — overrides everything below). Humans grade triage accuracy by applying one of these labels at close time:

   • triage-correct → verdict is VERIFIED-CORRECT; evidence is "human-applied triage-correct label". Skip steps 1–4.
   • triage-wrong → verdict is VERIFIED-INCORRECT; evidence is "human-applied triage-wrong label" (plus any human comment quoting the real RCA). Skip 1–4.
   • triage-transient → verdict is TRANSIENT; evidence is "human-applied triage-transient label". Skip 1–4.

   If none of the three labels is present, fall through to steps 1–4 below. See .claude/triage/LABELING.md for the labeling convention.

1. Re-read the issue body (failing tests, stack traces, job IDs) and ask: does the nightly agent's RCA actually explain that failure data? If the body shows AIMET quantize jobs failing and the agent said "AIMET regression," confirm — but only on top of additional evidence below. Plausibility ≠ correctness.

2. Read every human (*_QCOM) comment in full. If any human disagrees with or amends the agent's RCA, the verdict is VERIFIED-INCORRECT, regardless of who closed it.

3. Cross-reference fix PRs (from Call C). Match by:

   • PR body / title / commit message references the issue number
   • PR title mentions the failing model or test name
   • PR merged within 7 days of issue creation When a match exists, the PR's author/team is the strongest signal for whether the agent's named team was right.

4. Check the timeline for re-assigns. If the first assignee re-routed to a different team, the agent's routing was wrong; verdict is at best PARTIAL.

Rubric (positive evidence required for CORRECT verdicts):

Hard rule (the bias-breaker): if you cannot cite a specific piece of evidence — a fix PR number, a human-comment quote, or a timeline event — supporting a VERIFIED-CORRECT verdict, the score MUST be UNVERIFIED. Better to under-claim accuracy than to inflate it.

Build a summary table in working memory:

| Issue | Agent Team | Agent Cause | Evidence (PR # / human-comment quote / timeline event) | Score |

The Evidence column is mandatory and must appear in the final PR body too. A row without evidence is not VERIFIED-CORRECT.

2b. Scorecard accuracy

Scorecard issues have no single "fixer" — multiple regressions, multiple teams, no clean ground truth. Score only on the one falsifiable signal: did the regressions the agent labeled SUSTAINED actually persist into the next scorecard run?

Check the issue's labels first (highest-priority signal, same convention as Step 2a):

• triage-correct → score SUSTAINED-CORRECT; evidence is "human-applied triage-correct label". Skip the SUSTAINED-reappearance check below.
• triage-wrong → score SUSTAINED-WRONG; evidence is "human-applied triage-wrong label" (plus any human comment for context).
• triage-transient → score NO-DATA (or TRANSIENT); evidence is "human-applied triage-transient label".

If none of the three labels is present, fall through to the SUSTAINED-reappearance heuristic below.

For each scorecard issue from last week (created LOOKBACK_DAYS+7 to LOOKBACK_DAYS+1 days ago), find the matching scorecard issue from this week (created within LOOKBACK_DAYS days). For each (model, device, runtime, precision) tuple the agent flagged SUSTAINED last week, check if it appears as a regression this week.

Cluster quality and team routing are NOT scored by the SUSTAINED heuristic — too subjective without human follow-up. A human-applied triage-wrong label IS sufficient to score SUSTAINED-WRONG, regardless of the reappearance math, because the human is the higher authority.

Build a scorecard summary table — include an Evidence column citing the label (if any) or the reappearance ratio (if not):

| Last Week Issue | This Week Issue | Sustained Predicted | Sustained Confirmed | Evidence | Score |

2c. Identify KB updates needed

• New nightly error patterns: errors not matched by any existing pattern in error-patterns.md
• Nightly misrouted patterns: existing rules that led to wrong team suggestion
• New nightly examples: correct triage of non-obvious cases (add to examples.md)
• Team changes: fixers not listed in teams.md (add, never remove)
• New historical patterns from triage-wrong closures: for any issue graded VERIFIED-INCORRECT in Step 2a, extract the real root cause from the human comment or the fix PR and add a row to the appropriate section of historical-patterns.md (Transient, Workbench Service, Dependency Breakage, Compiler/Quantizer Bug, QAIHM Bug, External Contributor Bug, etc.). This is how last week's wrong triage becomes next week's known pattern. Only add concrete signatures the human evidence actually names (missing binary, specific dep version, specific import chain) — no speculation, no restatement of the agent's disproven guess.
• New scorecard signatures: regression patterns not matched by scorecard-patterns.md (add to the appropriate "Performance Regression Signatures" or "Numerics Regression Signatures" subsection)
• Confirmed flaky combos: (model, device, runtime) tuples that the agent labeled FLAKY two weeks in a row → add to "Known Flaky Model/Device Combos" table in scorecard-patterns.md (currently a placeholder)

If no updates are needed (all triage correct, no new patterns), skip to Step 4 with a "no updates" PR that just reports the accuracy scores.

Step 3: Apply Updates (~8 tool calls)

1. Create branch:

   git checkout -b auto/kb-weekly-update-$(date +%Y-%m-%d)
   

2. Edit the KB files using the Edit tool. Editable files:

   • error-patterns.md, teams.md, runtime-guide.md, examples.md — nightly KB
   • historical-patterns.md — recurring failure patterns; new rows come exclusively from triage-wrong closures where the human comment or fix PR identifies a specific, concrete signature (see Step 2c)
   • scorecard-patterns.md — scorecard KB (signatures + Known Flaky Combos table)

   Rules:

   • Only ADD or MODIFY — never delete without strong evidence
   • Keep edits minimal and targeted
   • Flag uncertain additions with "(low confidence — verify)"
   • For new error patterns: add to the appropriate confidence section
   • For new examples: append after the last numbered example
   • For misroute corrections: update the routing rule AND add an anti-pattern example
   • For new historical patterns: append a row to the section whose header matches the failure category (e.g. an ImportError from a missing system binary goes under "External Contributor Bug" or "QAIHM Bug" depending on whose code triggered the import; a compiler-side crash goes under "Workbench Compiler / Quantizer Bug"). Cite the tetracode issue number in the Occurrences column.
   • For new scorecard signatures: add to the right subsection (Performance / Numerics / Deployment) with Signal/Confidence/Team columns matching the existing format
   • For confirmed flaky combos: replace the (initially empty) placeholder row in scorecard-patterns.md with real entries the first time, then append after

3. Commit + push in ONE sequence:

   git add .claude/triage/
   git commit -m "Weekly KB update: {N} new patterns, {M} corrections
   
   Nightly triage accuracy: X/Y correct ({pct}%)
   Scorecard sustained accuracy: A/B confirmed ({pct}%)
   {one-line per notable change}"
   git push -u origin auto/kb-weekly-update-$(date +%Y-%m-%d)
   

Step 4: Create PR (~3 tool calls)

gh pr create --repo $REPO \
  --title "[Auto] Weekly triage KB update — $(date +%Y-%m-%d)" \
  --body-file /tmp/pr-body.md

Write /tmp/pr-body.md first with this format:

## Weekly Triage Knowledge Base Update

**Period:** {start_date} to {end_date}
**Nightly issues reviewed:** {count}
**Nightly triage accuracy:** {verified_correct}/{total} VERIFIED-CORRECT, {unverified} UNVERIFIED, {verified_incorrect} VERIFIED-INCORRECT, {transient} TRANSIENT, {missed} MISSED
**Scorecard issues reviewed:** {count}
**Scorecard sustained accuracy:** {confirmed}/{predicted} ({pct}%)

> Verdict reminder: VERIFIED-CORRECT requires a fix PR or human comment confirming
> the RCA. Silent closures default to UNVERIFIED — that is not a failure of the
> agent, only an absence of ground truth. Do not inflate the accuracy number by
> upgrading UNVERIFIED rows.

### Nightly Accuracy Report
| Issue | Agent Team | Agent Cause | Evidence | Score |
|-------|-----------|-------------|----------|-------|
| ... | ... | ... | "fix PR #NNNN by `team-X`" / "human comment: `quote`" / "closed without fix or comment" | ... |

### Scorecard Sustained Accuracy Report
| Last Week Issue | This Week Issue | Predicted | Confirmed | Evidence | Score |
|-----------------|-----------------|----------:|----------:|----------|-------|
| ... | ... | ... | ... | "label: `triage-correct`" / "5/6 SUSTAINED reappeared" / "no match this week" | ... |

### Changes Made
- {bulleted list of what changed and why, with links to issues/PRs}

### Misroutes Corrected
- {nightly misroutes, or "None this week"}

### Soft Observations (Scorecard)
- {non-scored notes about cluster quality or team routing — for human review only}

> Auto-generated by Breeze AI weekly KB update agent.
> Please review before merging.

Rules

• 50 action limit (each tool call = 1 action) — if approaching 45 actions, wrap up immediately (commit what you have)
• NEVER remove existing error patterns, team entries, or scorecard signatures — only add or modify
• Always create a PR for human review
• Batch gh calls — never loop over issues one-by-one
• If no nightly AND no scorecard issues this week → create a minimal PR noting "No issues to review"
• If only one of (nightly, scorecard) had issues, run that section and note "No {other} issues this week" in the PR
• If tetracode access fails → fall back to reading commit comments on $REPO
• Keep PR description under 10,000 characters
• Include links to original issues and fixing PRs for verification
• Scorecard cluster quality and team routing are NOT scored — only the SUSTAINED prediction is falsifiable