# Race Audit All > Run all nine LLK hazard audits (mmio-race, reconfig-stall, cfg-word-overlap, semaphore-handshake, mailbox-sync, dataflow-cb-sync, srcreg-bank-sync, noc-sync, instruction-latency) across four synchronization surfaces, and add a cross-class JOIN pass that catches emergent races no single audit can see — where one audit's verdict is "safe because ". Use for a full hazard sweep of an LLK change, or before merging anything touching config writes, reconfig/uninit, inter-thread/cross-core sync, the SrcA/SrcB-Dst data path, or hand-written instruction sequences. - Skill: `tenstorrent/race-audit-all` (Agent Skill) - Install (CLI): `npx skillmds@latest add tenstorrent/race-audit-all` - Raw SKILL.md: https://api.skillmd.com/api/skills/tenstorrent/race-audit-all/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: tenstorrent (https://skillmd.com/u/tenstorrent) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tenstorrent/race-audit-all --- # /race-audit-all — Orchestrated LLK race sweep + cross-class synthesis ## Purpose The per-class audits each cover one hazard mechanism. Most real bugs are category-local and any one audit finds them. But some are **emergent**: each audit individually says SAFE because its safety *depends on an invariant that lives in another audit's domain*, and no single audit verifies the join. This skill runs all nine and adds a **JOIN pass** that discharges those cross-references — without ever losing what the individual audits found. The nine sub-audits span **four synchronization surfaces** (not just cross-thread races — the suite also covers cross-core, RISC↔Tensix, and intra-thread micro-architectural hazards). Run each fully — see its own SKILL.md: - **Cross-thread (shared backend state):** - `cfg-word-overlap-audit` — two threads write the same 32-bit CONFIG word. - `reconfig-stall-audit` — config rewrite without draining the consuming execution unit. - `semaphore-handshake-audit` — inter-thread semaphore/mutex protocol (incl. SEMINIT-vs-usage). - `srcreg-bank-sync-audit` — SrcA/SrcB `AllowedClient`+bank-flip handshake (unpacker↔Matrix Unit) and shared-once Dst/LReg overwrite. - **RISC↔Tensix ordering:** - `mmio-race-audit` — RISC MMIO write vs Tensix-instruction/MOP/replay ordering; also `mop_sync`/`tensix_sync` drains (incl. OVER-SYNC/REDUNDANT perf findings). - `mailbox-sync-audit` — RISC↔RISC mailbox FIFO handshakes (push/pop balance, call-count symmetry, fence ordering caveat). - **Cross-core (NoC):** - `dataflow-cb-sync-audit` — circular-buffer producer/consumer credits (reserve/push/wait/pop balance, data-before-credit ordering, capacity, remote CBs). - `noc-sync-audit` — raw `noc_semaphore_*` + barrier data-before-signal ordering and multicast fan-out, plus the read-side / exit / coherency half of the same surface (an inbound read consumed before its read-barrier, a non-posted atomic left in flight at kernel exit, and on Blackhole a hand-rolled L1 poll missing `invalidate_l1_cache`) — the non-CB half of dataflow. - **Intra-thread (micro-architectural):** - `instruction-latency-audit` — pipeline result-latency / NOP padding on hand-written instruction sequences (compiler-grounded, arch-divergent). ## Recall preflight — run the `llk-audit` tool once, up front (augmentor, not a verdict) Before fanning out, run the deterministic recall tool for all its checks in one parse pass; it feeds every sub-audit a complete known-pattern worklist over one shared fact base (which is also what makes the JOIN a lookup rather than a schema reconciliation): tt_metal/tt-llk/.claude/tools/llk-audit/run.sh # Run this from the TT-METAL REPO ROOT — the path above is relative to it, and it is # the usual session CWD. (In a session rooted at tt-llk itself, drop the # `tt_metal/tt-llk/` prefix; from anywhere else, prefix an absolute repo path.) # Invoke it BY PATH and do not `cd` first: loading a skill does NOT change the shell's # CWD, so a bare `cd .claude/...` fails from the repo root. No `cd` is needed anyway — # run.sh chdirs to its own directory, so where you invoke it from changes nothing. # CONSEQUENCE: every PATH you hand it (an out_dir, `LLK_KT_LOG`) must be ABSOLUTE — # a relative one resolves against the tool dir, NOT your CWD. Use `$PWD/`. # PR-scoped sweep: add --changed [BASE] (default main) to scope every check to files changed vs BASE. # out/audit..json -> .checks[{mmio-race, cfg-word-overlap, # semaphore-handshake, reconfig-stall, # srcreg-bank, mailbox-sync, cb-sync, noc-sync, # noc-atomic-exit, noc-read-barrier, noc-l1-invalidate}] # (cb-sync + the four noc-* checks are committed + deterministic but need a # KERNEL fact base to yield findings — over tt-llk all five are trivially # empty; see kernel tier. mailbox-sync does yield its in-tree surface.) Hand each sub-audit agent its check's `findings[]` as the pre-enumerated worklist, and instruct it to **widen beyond the tool** per that check's `blind_spots` (the tool recalls KNOWN patterns only — the agents must still hunt the unknown). The tool ships committed deterministic checkers for **8 of the 9** classes — all but `instruction-latency` (its surface is the SFPU files clang can't parse + its verdict needs the out-of-tree pinned `sfpi-gcc` latency table → fully LLM-driven). Of the 8, **`cb-sync` and the `noc-sync` class (4 checkers: `noc-sync`, `noc-atomic-exit`, `noc-read-barrier`, `noc-l1-invalidate`)** only produce findings when fed a **kernel fact base** (the committed kernel tier's capture — see *Full-audit kernel tier* below); over the tt-llk fact base all five are trivially empty, and without a capture run their kernel surface stays LLM-driven (each skill's ttnn-widened grep). `srcreg-bank` recalls the dvalid control points, the raw-`SETDVALID`-on-BH flag and both halves of the Dest->Src gate (`DEST2SRC_*` flags the math side, which is the corruption half; on the unpack side `DUMMY_PUBLISH_SERIALIZING` is a throughput/parity candidate, while `DUMMY_PUBLISH_SETDVALID_UNSEQUENCED`, `DUMMY_PUBLISH_BOTH_BANKS_WAITLIKE` and `DUMMY_PUBLISH_PACKED_WAIT_WRONG_ARCH` are real encoding defects — the last a re-introduction guard, since only Wormhole defines those constants today) — but not the bank-flip lockstep verdict — and `mailbox-sync` recalls only the IN-TREE mailbox surface — mailbox use in ttnn/models kernels (one-to-one channels and fan-outs) is covered by the skill's ttnn-widened grep unless a kernel capture is run — so both still need heavy LLM widening per their `blind_spots`. The tool is **advisory**: it never clears a class, and its silence is "no new *known-pattern* instance," not "no bug". **If `run.sh` exits non-zero, the sweep continues manually — a failed tool is not a blocked audit** (the sub-skills already say "if unbuilt, proceed manually"; the orchestrator must too). It refuses rather than emitting a false all-clear in four distinct cases, and they mean different things: the **extractor won't build** (no Clang/LLVM ≥ 18) or **no facts extracted** → every class is LLM-driven this run, say so; a **`--changed` base that doesn't resolve** → fix the ref, never accept the empty scope; a **kernel-tier failure** under `--full-jit` → the in-tree findings above it still stand. In all four, report the tool status alongside the findings so no reader mistakes "the tool didn't run" for "the tool found nothing". **Coverage caveat — attend to `parse_errors` (a per-run partial-coverage signal).** The envelope's `parse_errors` is `>0` on a normal run: the SFPU-heavy files that use `sfpi::` types don't fully parse (expected — see `out/parse.log`), so writes in those files are ABSENT from the fact base. That is a *known, expected* partial-coverage bound, NOT a clean full-coverage guarantee — treat any class whose real surface is SFPU-side as tool-under-covered there and widen manually. **But before trusting a low/zero finding count, scan `out/parse.log` for a header that should NOT fail (a NON-SFPU file):** an unexpected parse failure there is a silent coverage hole (that class's writes in that file were never analyzed → its "0" is meaningless). This is the consuming-side half of the never-false-all-clear contract. **Read the envelope's `degraded` list too — it is the tool's own "I did not analyze this" channel.** `cli.py` records there every surface it could not analyze this run (empty/near-empty fact base, unreadable cfg defines, a `--changed` file that parsed to 0 facts, a kernel-tier capture coverage hole, uncertain attribution under a partial parse), and `run.sh` prints it as `*** DEGRADED — NOT a clean all-clear ***`. A non-empty `degraded` makes a low/zero count **unanalyzed, not clean**: carry each note into the report, treat the named surface as LLM-only for that class, and never let a degraded run close a class. **Tool-drift contract — detect a stale registry, surface it, offer the fix (do NOT silently edit).** The deterministic tier is only as complete as `llkaudit/registry.py`'s name→meaning tables. When the codebase adds a sync-relevant API the registry doesn't yet know — a new `CircularBuffer`/`Noc` method, a `noc_async_*`/`noc_semaphore_*` variant, a new cfg-write helper or `TTI_*` cfg instruction, a new mailbox call — the checkers **under-recall silently**: fewer findings, no error, exit 0. A stale registry does not look broken; it looks *clean*. So during any audit (especially a full sweep) treat these as **tool-drift tells** and act on them: - an `UNRESOLVED` bucket rising sharply in the checks that HAVE one — cfg-word (`UNRESOLVED`) and mailbox (`UNRESOLVED_ENDPOINT`) surface inputs they couldn't key; a jump means a new write/endpoint shape the registry doesn't classify; - the kernel-tier **coverage ledger** showing TUs it could not translate/parse, or did not analyze cleanly for another reason — any row whose status LEADS with a hole marker (`PARSE-FAIL`, `PARSE-HOLE`, `EMPTY-OUT`, `EXEC-FAIL`, `SKIP-*`, `HOST-LEAK`, `NON-KERNEL-CMD`). Such rows are excluded from the ledger's own `ok/N TUs parsed` headline, and their COUNT is passed into the audit JSON's `degraded`. A *capture* hole (the checkers never saw those kernels), distinct from a classification gap but equally a silent miss; - **the LLM tier finding a real sync site the deterministic worklist omitted** — the cleanest tell, and the ONLY one for **cb-sync / noc-sync**, which have no `UNRESOLVED` bucket (a CB/NoC object has many non-flow-control methods, so an unknown method can't be flagged deterministically without over-firing) — so a new `CircularBuffer`/`Noc`/`Semaphore` flow-control method shows up only as a site the LLM found that the tool's candidate list didn't. **Before calling anything drift, check `KNOWN_GAPS.md`** (next to `registry.py` — the tool's single canonical list of *deliberately deferred* gaps, each with its risk class, live-today count, and why the obvious fix was refused). A gap listed there is real but consciously not fixed, usually because the naive fix trades recall for false-positives — so do **not** re-file it as newly-discovered drift: cite its `L#`/`X#` and move on. If this sweep establishes a NEW gap the user chooses to defer, record it **in `KNOWN_GAPS.md`**, not in the run report (a report rots; the ledger is read by the next run). On any such observation the ledger does not already own, **name the stale entry** (which `registry.py` table + which checker under-recalls, and the concrete API missed), tell the user the deterministic tier is drifting, and **offer to add it**. Apply the enhancement **only if the user allows** — never silently. The fix is almost always a one-line `registry.py` table entry (the single declarative edit point) plus a hermetic test in `tests/test_checks.py`; rarely a checker change; **never** the C++ extractor unless a genuinely new fact family is needed. Ground the new entry against the authoritative header (e.g. `circular_buffer.h`, `noc.h`, `dataflow_api.h`, `ckernel_ops.h`) per the source ladder, keep the ground-truth counts fresh, and follow the commit-twice pre-commit discipline. Then re-run recall so the sweep reflects the widened tables. This keeps the tool a living superset of the codebase instead of decaying into a false all-clear as the APIs move. **Drift runs both ways.** Under-recall (`CAP-REDUCTION`) is the more serious class, but a checker can also **over-report** (`FALSE-FLAG`) — a flush or credit form its registry doesn't yet recall, or a receiver-type text heuristic admitting a non-CB/non-NoC object. So a tool `findings[]` entry is a **candidate, not a verdict**: confirm it at the site before it enters a report. Dismissing an unconfirmed *tool candidate* with shown evidence is **not** a monotonic-contract downgrade — that rule protects the sub-audits' own verdicts (below), not the recall tool's pre-verdict worklist. ## Full-audit kernel tier (opt-in) — the committed JIT capture for cb-sync / noc-sync / noc-atomic-exit / noc-read-barrier / noc-l1-invalidate / mailbox-sync The `cb-sync`, `noc-sync`, `noc-atomic-exit`, `noc-read-barrier`, `noc-l1-invalidate`, and `mailbox-sync` **checkers are committed and deterministic** — but their kernel surface lives in **JIT-compiled kernels OUTSIDE tt-llk** (`ttnn/`, `models/`, `tt_metal/hw/inc/api`), which have no static compile database the in-tree fixed-flags parse can reach. So the checkers are always present; what's missing at rest is a **kernel fact base** for them to run over. The `kernel_tier/` module (committed in-tree) produces that fact base; the only runtime-dependent step is the capture RUN itself. > **What is durable vs runtime-dependent (the clean split):** > - **DURABLE (committed, in the tool):** the `cb-sync` / `noc-sync` / `noc-atomic-exit` / `noc-read-barrier` / `noc-l1-invalidate` / `mailbox-sync` > checkers themselves — plain Python over a fact base, unit-tested, no JIT hook — > AND `kernel_tier/{capture.py,bootstrap.sh,MANIFEST}`, the capture pipeline. > Over the tt-llk fact base **cb-sync and all four noc-* checks are trivially > empty** (no cb/noc sites there) while **mailbox-sync yields its small in-tree > surface** (the MATH→UNPACK dst_index pair + debug endpoints); fed a kernel fact > base all six emit real candidates over the outside-tt-llk kernels. > - **RUNTIME-DEPENDENT (each sweep):** the capture RUN — producing a build log > that carries the JIT compile commands, then translating each RISC-V-GCC command > to clang. This needs a build log or a live runtime, and the GCC→clang > translation is the fragile part — isolated in `capture.py` with an honest > coverage ledger (untranslatable TUs are listed, never silently dropped). No > `ccwrap` / compiler-wrapper and no `jit_build` patch: capture is just a log scrape. **CURRENT STATE:** the module is **committed and in-tree**. `run.sh --kernel-tier-status` prints **`available`**, and `run.sh --full-jit` runs the in-tree audit **and then** `kernel_tier/bootstrap.sh`. It never captures silently: bootstrap needs either a pre-captured log (`LLK_KT_LOG`) or permission to run a workload (`LLK_KT_WORKLOAD` [+ `LLK_KT_CLEAR_CACHE=1`], which needs a device/sim). **When to offer it — gate on BOTH conditions:** 1. **Mode:** the user asked for a **full / exhaustive** sweep (NOT a diff/PR-scoped `--changed` run). A diff/PR run **never** prompts — the fast path stays clean. 2. **Runtime:** the capture needs a build log or a device/sim. If neither is available, do **NOT** silently improvise. Proceed with the in-tree sweep and **tell the user precisely what that means for cb-sync / noc-sync / noc-atomic-exit / noc-read-barrier / noc-l1-invalidate / mailbox: they are NOT tool-recalled *over kernels* this run, but they are STILL AUDITED — LLM-driven via each skill's (ttnn-widened) grep + reasoning + ISA docs. "Not tool-recalled" ≠ "left out."** The only thing forgone is the extra deterministic candidate list (grep-fidelity recall, so macro-wrapped/aliased calls and cross-kernel pairing may be missed — state that as the coverage bound). **Cost honesty before running it:** capturing by running a workload needs a **runtime** (hw/sim), runs a workload build (minutes, and `LLK_KT_CLEAR_CACHE=1` forces op-kernel recompilation), and gives **periodic-sweep-grade** coverage that is **complete only over the kernel variants actually exercised** (a clean result must never read as "all kernels covered"). **How to run the capture** (the module is already committed — you only RUN it): 1. **Get a build log** carrying the JIT compile commands — either `TT_METAL_LOG_KERNELS_COMPILE_COMMANDS=1 > $PWD/build.log 2>&1` captured on hardware once (then audit offline), or let bootstrap run the workload for you. 2. **Run the tier** — same `run.sh` path as the preflight above, from the repo root, and an **absolute** `LLK_KT_LOG` (per the preflight's path rule; a relative one resolves against the tool dir and fails as "not found"): `LLK_KT_LOG=$PWD/build.log tt_metal/tt-llk/.claude/tools/llk-audit/run.sh --full-jit` (log path), or the run-a-workload path: `LLK_KT_CLEAR_CACHE=1 LLK_KT_WORKLOAD='' tt_metal/tt-llk/.claude/tools/llk-audit/run.sh --full-jit`. bootstrap → `capture.py` (scrape → GCC→clang translate → `llk_extract` per kernel → merged fact base) → cb/noc/read/atomic/l1/mailbox over it. 3. **Coverage is emitted automatically:** `kernel_coverage..txt` lists every TU as parsed or with a leading hole marker (`PARSE-FAIL`, `PARSE-HOLE`, `EMPTY-OUT`, `EXEC-FAIL`, `SKIP-*`, `HOST-LEAK`, `NON-KERNEL-CMD`) — no silent caps. Holes reach the audit JSON's `degraded`, so *some* kernels failing can't be hidden by others succeeding; an empty fact base makes bootstrap exit non-zero outright rather than emit a false all-clear. **Read `degraded` before treating 0 findings as clean.** 4. **Merge:** merge/dedup the kernel-tier candidates with the in-tree findings. Everything lives **inside tt-llk** (`.claude/tools/llk-audit/`), so there is zero permanent footprint outside tt-llk. Whatever the kernel tier surfaces is **candidates** (augmentor) — the data-before-credit ordering, cross-kernel producer↔consumer pairing, and mailbox call-count-symmetry/ordering **verdicts stay with the sub-audit skills**. ## The monotonic contract (non-negotiable — this is what makes the sweep a true superset) A naive "run them + concatenate" can catch *less* than the audits alone (summarization loss, dedup collapse, over-resolution). To prevent that, the JOIN is **additive-only**: 1. **Preserve every per-audit finding verbatim.** The output *includes* all nine raw reports (full finding lists, not summaries). Nothing is dropped, merged-away, or reworded. 2. **The JOIN may only ADD findings or ESCALATE severity.** It must never silently delete or downgrade a per-audit verdict. 3. **No silent downgrades.** If the JOIN verifies a cross-reference and believes a flagged item is actually safe, it attaches an *annotation* next to the original flag with the shown evidence — it does **not** replace the flag. Default to keeping the flag; never upgrade "probably safe" to SAFE without proof at that exact site. 4. **No summarization at the fan-out boundary.** Sub-audits return their full enumerations + verdicts; the JOIN reasons over those, not over compressed digests. If you fan out to agents, instruct each to return the complete finding list (and its candidate count) so the JOIN can re-judge dismissed sites if a cross-reference implicates them. 5. **No cap without a closer.** If coverage is bounded anywhere (top-N, sampling, agent budget, a file left unopened, a verdict resting on an unconfirmed claim), `log`/state it **AND treat it as a work item, not a terminal state**: an exhaustive run must spawn a gap-closer for each bound and **loop closers until the bound is gone** (see *Exhaustive coverage — file manifest + ledger* below). A bounded sweep must never read as exhaustive, and "I sampled N of M / deferred the experimental files" is an **incompleteness that blocks the done-verdict**, not an acceptable caveat. ## Method 1. **Run all nine audits** (faithful execution — invoke each skill or run its deterministic enumeration; don't approximate). Collect each one's complete finding list with per-finding `file:line`, verdict, and — critically — its **stated assumption** for every SAFE/LATENT verdict (the "safe because …" clause). 2. **Build the cross-reference worklist.** Extract every verdict whose safety is conditional on another class's invariant. The known seams (starting set — not exhaustive): | Audit says… | …safe because (other-class invariant) | JOIN must verify | |---|---|---| | `cfg-word-overlap`: shared SrcA/SrcB ALU-format word is LATENT | "pipeline semaphores keep the reconfig from overlapping the other thread's op" | the *specific* format RMW sits inside the region that `semaphore-handshake` proved ordered — same semaphore, write between wait and post/get, on every branch | | `mmio-race`: MMIO config write SAFE | "a semaphore / STALLWAIT(TRISC_CFG) orders it before the consumer" | `semaphore-handshake` shows that semaphore is balanced+init'd AND the MMIO store is sequenced relative to the post; or the stall's condition actually covers the consumer | | `reconfig-stall`: per-thread drain present (e.g. `STALLWAIT(STALL_CFG, PACK)`) | (drains *this* thread's unit only) | does another **thread** write the same word? → hand to `cfg-word-overlap`; a per-thread drain never excludes a cross-thread writer | | `semaphore-handshake`: semaphore protocol SAFE | (verifies counting, not payload) | which config words/dest/src rely on this semaphore for mutual exclusion? → confirm each such write is actually inside the ordered window | | `mailbox-sync`: mailbox handshake SAFE | "the memory the mailbox value refers to is ready, and all threads reach the mailbox handshake equally (including hand-written mailbox_write in ttnn/models kernels)" | the referenced memory (L1 tile, dest offset) is ordered-ready — a plain `fence` does NOT order a mailbox write against a prior store to a different region (a no-op on WH; on BH it drains the store queue but not to *processed*), so cross with `mmio-race`/memory-ordering AND hand the "is the CB page ready?" half to `dataflow-cb-sync`; and the call-count symmetry holds on every branch (same control-flow that `semaphore-handshake` balance depends on) | | `dataflow-cb-sync`: CB credit SAFE | "the page write is ordered before the credit, and reserve/wait gates the access" | the data-before-credit barrier (NOC flush before `cb_push_back`) is present → cross with `mmio-race`/NOC ordering — this holds for the Metal 2.0 `DataflowBuffer` object form too, whose credit quartet IS recalled but whose per-buffer `dfb.write_barrier(noc)` is NOT recognized as a flush, so confirm the flush at the site instead of trusting a no-flush candidate; the address `mailbox-sync` sends (over its directed tile-address channel) derives from `fifo_rd_ptr` gated by this `cb_wait_front`; and `tile_regs_*` interleaving is `semaphore-handshake`'s `MATH_PACK` | | `mmio-race`: MMIO-vs-MOP write SAFE | "a `mop_sync()`/`tensix_sync()` drains it" | the drain provably covers the consumer at the site (right primitive, every path, cross-call window). ALSO: is the drain heavier than needed (OVER-SYNC) or unnecessary (REDUNDANT)? → perf finding, never suppresses the race verdict | | `srcreg-bank-sync`: SrcA/SrcB bank handoff SAFE | "the FPU op waits for `AllowedClient`, and Dst/LReg is ordered" | bank-flip is lockstep on both sides; the Dst/LReg half rides `MATH_PACK`/`mutex::SFPU` → hand that half to `semaphore-handshake`; single-thread ownership of the bank state holds. **The `AllowedClient` premise fails for `MOVD2A`/`MOVD2B`** — they *write* Src, so the Wait Gate does not cover them; verify the explicit `STALLWAIT` drains the FPU pipe AND names the bank-valid condition of the register the move writes, that no Src bank flip is issued between that stall and the move, and that the paired unpack-side dummy publication waits on the unpacker's own bank (`srcreg-bank-sync` check 5) | | `noc-sync`: cross-core credit SAFE | "the remote write is flushed before the credit, and the wait count matches the fan-out" | a WRITE credit (`set_remote`/`set_multicast`/`relay_*` — **bare `noc_semaphore_set` is a LOCAL reset store, not a credit**) same-NoC/VC/dest is ordered by issue-order **only when payload and credit target the same memory kind**: per `Ordering.md`, an L1 write followed by a write to an **MMIO** address (a stream/config register — the `noc_inline_dw_write` shape) has **no** ordering guarantee and the MMIO write can race ahead, so a register-target credit needs a barrier like an atomic one; an ATOMIC credit (`noc_semaphore_inc`/`inc_multicast`/remote `up`) needs the payload write **committed** — `noc_async_write_barrier` (ACK), not a bare `writes_flushed` (departure only, per `data_movement_doc/general/posted_writes.md`). Do NOT clear a flush-only atomic (the checker tags it `FLUSH_NOT_BARRIER`, or `POSTED_FLUSH_ONLY` when the only preceding flush drains the posted-write counter — a no-op for a non-posted write/inc) as safe on a same-VC-unicast assumption — confirm against `/NoC/Ordering.md` + `posted_writes.md`. Cross with `dataflow-cb-sync` when the same buffer is also a CB page | | `dataflow-cb-sync`: credit-counter read SAFE | "the counter read observes the other side's update" | on **Blackhole** that is a *cache*-coherency claim, not a CB-API one: a HAND-ROLLED poll of a CB/credit word (anything not going through `cb_wait_front`/`cb_reserve_back`) needs `invalidate_l1_cache` — the invariant `noc-sync`'s L1-coherency lens owns → hand it there. Latent while the RISC L1 D-cache is off by default; a real hang once it is enabled | | `instruction-latency`: sequence SAFE | "the compiler scheduled the NOPs" / "Blackhole HW scoreboards it" | the code is actually sfpi-compiled (provenance lens), not raw `TTI_*`; and for BH the consuming insn is NOT in the freshly-derived `xtt_dynamic_bug` errata set — re-derive from the pinned `sfpi-gcc`, never a baked list | | any: "value-invariant / unit-idle / single-thread" | (assumption about another class's state) | re-confirm the assumption at the site with the other audit's lens | 3. **Discharge each cross-reference at its actual site** (read the code; don't trust the summary). Trace the one physical resource (a CFG word, dest/src bank, a semaphore) across *all* mechanisms that are supposed to guard it. A race exists when the guarantees, composed, leave a gap — even if each guarantee is individually valid. 4. **Emit** per the monotonic contract: nine raw reports untouched, plus a new **EMERGENT** findings section, plus any escalations (annotated). ## Verdict (combined) - **EMERGENT-RACE** — individually all-SAFE, but the join condition is unmet at a reachable site (e.g. format RMW moved outside the semaphore window; MMIO write not actually sequenced to its gating post). Real; report with the full cross-class chain. - **Per-class verdicts** — passed through unchanged from each sub-audit (SAFE / RACE / LATENT / HARDENING-GAP / INIT-BUG …). - **ANNOTATED-SAFE** — a per-class flag the JOIN cross-checked and believes is discharged; keep the original flag, attach evidence, mark for maintainer confirmation. Never silently resolve. - **UNCERTAIN — needs HW/owner confirmation** — the label mandated by *Never infer a NEGATIVE from a missing doc* (below) when no reachable authority resolves the point. It is a **surfaced, non-closed** finding: it counts in the totals, is neither RACE nor SAFE, and must never be silently promoted to either. A JOIN seam whose invariant could not be discharged lands here, not in EMERGENT-RACE. - **FALSE-POSITIVE (unreachable)** — a value-gated hazard ruled out by a host-side invariant (program factory / device-op validation / `constexpr`), per *Confirm reachability* below. Report it with the invariant that rules it out, so the next run doesn't re-raise it. ## Ground-truth freshness contract (non-negotiable) Two sub-audits depend on sources that are **dynamic** and must be consulted live each run, never from data baked into a SKILL.md: - `instruction-latency-audit` grounds in the **`sfpi-gcc` version this build is pinned to** (latency tables, `xtt_delay`, `xtt_dynamic_bug` errata) — re-derive every run; optionally diff the tip and flag divergence; mark coverage bounded if the pinned compiler can't be resolved. - All HW-semantics verdicts ground in the **tt-isa-docs MCP** (fetched live), not cached prose. Any instruction list / latency number / errata set appearing in a sub-skill is a *dated illustration*, subordinate to fresh derivation. The JOIN must not treat a stale baked list as authority. **ISA-precedence policy (applies to every sub-audit):** the live ISA doc (and, for `instruction-latency`, the pinned `sfpi-gcc`) **outranks** any rule, table, or example baked into a skill. When a live source **contradicts** a baked rule, do NOT silently resolve it — surface the conflict to the user and ask whether the baked rule should be overwritten, discarded, or kept; default to the live source. This holds whether a sub-audit is run standalone or inside this sweep. **Ground-truth source ladder (a SUPERSET of the sage agents' corpus; `assembly.yaml` deliberately excluded) — use every applicable source each run and combine them:** Authority is **per audit-class × arch**; never skip a source because it lacked coverage last time. HW facts are confirmed against an authoritative source, never extrapolated. **In-repo code is a first-class living source, but only for facts that ARE the code** (applies to every arch below): instruction existence/encoding — `tt_metal/tt-llk/tt_llk_/common/inc/ckernel_ops.h` (a `TT_OP_*`/`TTI_*` with an opcode = valid on that arch, regardless of ISA-doc coverage), and *what a call actually issues* — VC / cmd-buf / posted / barrier — `tt_metal/hw/inc/api/dataflow/dataflow_api.h` + the per-arch `tt_metal/hw/inc/internal/{tt-1xx/,tt-2xx/quasar}/noc_nonblocking_api.h`. Code is **NOT** authoritative for HW *semantics* — whether HW then orders/latches those transactions, latency, sampled registers, errata — that is the ISA doc's domain, and existence-in-a-header is never a reason to skip it; **nor** for the correctness of the audited kernel (self-certification is circular). A code **comment** is only an author's belief, never ground truth (e.g. an in-repo `#ifdef ARCH_BLACKHOLE` comment asserting BH reorders across cmd buffers was wrong — BH is `noc2axi`-ordered; ground ordering in `/NoC/Ordering.md`, not the comment). - **HW-semantics audits** (mmio, reconfig, cfg-word, semaphore, mailbox, srcreg, noc, dataflow-cb): - **WH B0 / BH A0:** **tt-isa-docs MCP** *or* **DeepWiki** — both serve the same `tenstorrent/tt-isa-documentation` corpus, either is fine — as primary ISA; **code** (per the code principle above) to cross-verify. **Fetch ISA pages by EXACT path (raw GitHub, or `gh api repos/tenstorrent/tt-isa-documentation/contents/`), NOT index search** — *both* index-backed routes (semantic doc search AND repo code search) silently miss real pages, and can return fully empty when the server's upstream read of the corpus fails, with no error to distinguish that from a genuine miss (a prior audit wrongly concluded `/NoC/Ordering.md` "did not exist" and inferred a race from the gap). A miss from either is **NO EVIDENCE**, never absence; canary-gate them per the *Source preflight* below. **WH/BH Confluence:** one canonical pointer — `1001357404` → **noc / dataflow-cb** HW-bug workarounds (WH/BH); open at audit (opaque ID, don't transcribe). - **Quasar:** **tt-isa-docs MCP** — *queried every run* (empty for QSR today, but coverage is expected; promote to primary the moment it lands; do **not** use DeepWiki for QSR — it has none); **Quasar Confluence HW pages** (internal/authenticated only) as the de-facto primary today — **don't rely on naïve CQL search; anchor to these canonical pages** (seed list, not exhaustive; run a freshness check before citing any — pages older than ~3 months get a staleness caveat, >9 months or undated must be re-verified against code/HW). These are **pointers** — open the page in Confluence at audit for its contents; do not transcribe page titles or page facts into reports or skills. The page IDs (kept opaque on purpose — open them at audit): - `1340276980` → grounds **cfg-word / mmio / reconfig** - `1613201604` → per-instruction semantics for any class - `586515553` → Quasar↔WH/BH arch deltas - `646217858` → **semaphore / noc** - `113017320` → **srcreg / semaphore** - `1256423592` → **instruction-latency** HW context - `84508873` → general arch overview - `627475207` → **noc / dataflow-cb** HW-bug + feature tracker (Quasar; historical/bring-up — verify against code at audit) Search beyond these when a topic isn't covered, but start here. If a seeded ID returns 404 (page deleted/recreated), re-find the page in Confluence by topic and update the ID. **BH-inference** only as a caveated last resort (never grounds or overturns a verdict). Where two sources carry a fact, cross-check and surface conflicts. - **instruction-latency (all archs):** the pinned **`sfpi-gcc` source** (`rvtt.md` / `sfpu-ops-{wh,bh,qsr}.h` / `rtl-rvtt-schedule.cc`) — **fetch it at the pin**; the compiled toolchain / a compile experiment is not a substitute. Secondary: tt-isa-docs `VectorUnit` (WH/BH HW latency) and the Quasar SFPU uArch Confluence page (QSR HW context). (Instruction existence/validity → `ckernel_ops.h` per the code principle above, never ISA-doc coverage.) - **dataflow-cb / noc:** the **dataflow API source** (per the code principle above — authoritative for *what a call issues*: VC / cmd-buf / posted / barrier) plus the per-arch ISA sources above. **NoC transaction ordering** grounds on the ISA `/NoC/Ordering.md` page (fetch by exact path per the WH/BH note; the WH page carries the same-source/dest/NoC/VC in-order rule, write→atomic having *no* ordering rule, and response/ack always reorderable). **Blackhole (re-check every run — a 404 is point-in-time, not permanent):** fetch `BlackholeA0/NoC/Ordering.md` each run; the moment it exists it becomes the authority and supersedes the fallback below. While it is absent (404 at time of writing), ground BH ordering on code + `1001357404` + the `noc2axi` ordering fact confirmed by the SOC/NoC team, marked a coverage caveat. Secondary OP-writer-level tier: the `tenstorrent/tt-low-level-documentation` repo `data_movement_doc/` (barrier taxonomy, posted-vs-nonposted, VC-vs-credit tension). Quasar adds the tile-counter / global-semaphore Confluence pages. **Ground-or-abstain (this is what makes a re-run a superset of any prior run):** ground each verdict against the *applicable* authorities at full strength; if **none** is reachable, **emit no verdict** for that point (label the coverage hole, e.g. `[Quasar: code-only]`) — never substitute a weaker basis (WH/BH extrapolation, a code comment, a compile experiment) for a verdict, and never let a weaker-grounded result overturn a stronger-grounded one. **Never infer a NEGATIVE from a missing doc (guards against the dominant false-positive class).** A missing ISA-doc page means *undocumented*, never *absent/invalid/unordered* — the corpus is an explicitly incomplete living document. So doc-silence must NOT become a confirmed `RACE`/`INVALID` verdict, and equally must NOT become `SAFE`. Before asserting any negative ("instruction invalid on arch Y", "ordering not guaranteed → race"): (1) first try to *resolve* it against the applicable authority in the ladder above (e.g. instruction existence → `ckernel_ops.h`; HW ordering → `/NoC/Ordering.md`) and take the resolved verdict if found; (2) if no reachable source resolves it, emit **UNCERTAIN — needs HW/owner confirmation** (a surfaced, non-closed finding), *not* a confirmed race and *not* safe. This is not a relaxation of conservatism: the item is still raised for confirmation (nothing escapes), it is only labeled by true confidence. A fabricated mechanism filed as "confirmed" (e.g. a "BH reorders cross-cmd-buffer → RACE" claim later refuted by the SOC team as `noc2axi`-ordered) is *anti*-conservative — each refuted confirmed-flag trains owners to dismiss the auditor, so real races later get waved off. **Record** which source + revision (ISA-doc rev / DeepWiki, Confluence page + date + freshness, sfpi-gcc commit) each verdict was grounded against, so runs are reproducible and comparable. (A stateless run cannot *recall* a prior one — to be a superset of a **specific** prior run, supply that run's report as an input and reconcile against it monotonically.) **Confirm reachability before flagging a value-gated race/deadlock (a positive verdict needs grounding just as much as a negative).** When a hazard depends on a specific runtime value — a loop count of 0, a dimension of 0, an empty-range iteration, an `== N` branch — the code path being statically present is NOT sufficient. Verify the triggering value is actually reachable given the op's **host-side invariants**: the **program factory** (compile-arg derivation, grid/shape math), the **device-op validation** (`TT_FATAL`, layout requirements — e.g. a TILE-layout requirement forces tile-aligned dims ≥ 32), and compile-time constants (`constexpr`). If a repo invariant makes the triggering value impossible, the finding is a **FALSE-POSITIVE (unreachable)** (the verdict label above) — or at most a latent "if this invariant is ever relaxed" note — never a confirmed race. A kernel cannot be audited in isolation from the factory that launches it; this is the same ground-in-the-authoritative-source discipline, with the factory + device-op validation as the source. Examples: a `Ht==0` deadlock ruled unreachable because the op `TT_FATAL`s TILE layout (so `Ht ≥ 1`); an off-by-`start_core_id` loop bound that is correct because `start_core_id` is `constexpr 0`. **Source preflight — emit this FIRST, before any audit work, and PAUSE for the user.** Before running any sub-audit, build a **source manifest** from the ladder above and **probe each source's reachability**, then present it and let the user decide. Render a table — *source · tier/role · arch+class served · reachable? (✓ / ✗ / N-A) · note* — covering at least: - **tt-isa-docs MCP** (WH/BH ISA; also queried for Quasar — expected empty today) — callable? **Probe with a CANARY, never a bare call.** It is a third-party proxy over the ISA corpus, and when its upstream read of that corpus fails it answers a *successful empty result* — "No code matches found" / "No documentation found", `fileUsed: "unknown"` — not an error, which is indistinguishable from a real miss and reads as "the page does not exist". So query the index for a term that CANNOT be absent (e.g. `Tensix`): empty canary ⇒ the **index is down**, mark the row `✗ [index down]`, and for the rest of that run treat every index miss as **NO EVIDENCE** (never absence — see *Never infer a NEGATIVE from a missing doc*). Report the two routes as SEPARATE rows, because they fail independently: **index search** (canary-gated) and **exact-path fetch** (raw GitHub / `gh api …/contents/`), which bypasses the proxy's corpus index and so survives its outages. Exact-path reachable ⇒ no verdict is bounded by a dead index. Both routes dead ⇒ WH/BH lose their primary ISA authority, so abstain per *Ground-or-abstain* — except where the ladder names another WH/BH source (the `1001357404` Confluence pointer for **noc / dataflow-cb**), which can narrow such a verdict but never replace the ISA page. - **DeepWiki MCP** (WH/BH ISA, equivalent corpus) — callable? Same proxy class, so **same canary rule + same `[index down]` label**; being the redundant route for the same corpus, it is only a coverage loss if tt-isa-docs' exact-path fetch is ALSO unreachable. - **Confluence / Atlassian MCP** (Quasar HW; internal-auth) — authenticated & reachable? - **sfpi-gcc source** (instruction-latency) — resolve the pin: `tt_metal/sfpi-version` gives an `sfpi_version` tag → that tag of `tenstorrent/sfpi` → its `gcc` submodule commit in `tenstorrent/sfpi-gcc`; confirm the files fetch (the exact three-step recipe lives in `instruction-latency`'s freshness contract — follow it there, don't re-derive). - **dataflow API source** (`tt_metal/hw/inc/api/dataflow/`) — present? - **code** (`tt_llk_*`) — present (≈ always). Then state, **per unreachable source, which verdicts w …(truncated)