Noc Sync Audit

/noc-sync-audit — cross-core NoC handshake correctness

Ground-truth precedence: the live ISA doc (tt-isa-docs MCP, fetched each run) outranks every rule, table, and example baked into this skill — treat those as dated illustrations. If the live ISA doc contradicts a baked rule here, do NOT silently proceed: surface the conflict to the user and ask whether the baked rule should be overwritten, discarded, or kept. Default to the ISA doc.

MANDATORY — before any verdict, read the shared grounding policy. The per-architecture source ladder (which docs to consult), the ground-or-abstain rule, and the Source preflight (list the sources you'll consult with their reachability + hierarchy, then PAUSE for the user) are defined once in race-audit-all → .claude/skills/race-audit-all/SKILL.md. Your FIRST action is to Read that file and follow its "Ground-truth source ladder", "Ground-or-abstain", and "Source preflight" sections — they are load-bearing: a verdict produced without them is ungrounded and MUST NOT be reported. If that file genuinely cannot be read, say so and abstain rather than proceed ungrounded. (If you were spawned by a race-audit-all sweep — your prompt already lists the confirmed sources — skip the Source preflight and do not pause; the orchestrator ran it once.)

Coverage — floor, not ceiling. The grep patterns and site lists in this skill are a seed, not an exhaustive enumeration. After running them, widen the search with full reasoning. The techniques here are illustrative examples, not the allowed set — use any approach your reasoning suggests, including ones not listed: e.g. semantic search (by behavior/effect, not just token), resolving macros / wrappers / typedefs / indirection the literal pattern can't match, following the call graph to callers and callees, and diffing the WH/BH/QSR variants to catch a site present in one arch and missing in another. If you can find a hazard, primitive, or site the encoded patterns don't cover — by any means — pursue and report it; do not clamp a stronger analysis to this list or to these techniques. State any residual coverage gaps explicitly (no silent caps).

Execution — parallel by default. When enumeration yields more than a few sites/files, fan out concurrent Agent calls by default (one per file/subsystem, a fresh context each), saturating the available concurrency (~10–16 at once); go inline only for a trivial set. The per-file fan-out described under Thoroughness is the default, not an exhaustive-only option. The cross-referencing/synthesis of results stays sequential (it must follow the per-unit findings). The heavyweight Workflow tool still requires explicit multi-agent opt-in — it is the opt-in exhaustive tier, not the default. Don't over-spawn a tiny diff.

Persisting results — single writer, incremental. Agents only return their findings; they never write a shared file (no concurrent-write clobbering). If findings are persisted to a file, the orchestrator/caller is the sole writer and appends each wave's returns as they arrive — incremental, never only-at-the-end — so an interrupt preserves every completed wave's findings.

Recall preflight — run the tool first (augmentor, not a verdict)

Before enumerating, run the deterministic llk-audit NoC checkers for a complete known-pattern worklist. This class is backed by four committed checkers — noc-sync (write→credit ordering), noc-atomic-exit, noc-read-barrier, noc-l1-invalidate:

# from the tt-metal repo root; do NOT cd into the tool dir (run.sh self-locates)
tt_metal/tt-llk/.claude/tools/llk-audit/run.sh <arch> --full-jit   # kernel tier runs all four

All four are empty over the tt-llk headers — their surface is JIT-compiled kernels (ttnn/models), so they only emit findings when fed a KERNEL fact base via --full-jit (the on-request capture; runbook in race-audit-all). Treat each checker's findings[] as a floor, not a ceiling — widen per its blind_spots (each names its own gaps: e.g. noc-read-barrier over-matches non-issuing read helpers and doesn't verify buffer identity; noc-atomic-exit is scoped to kernel_main). If the kernel tier isn't built, the checkers report nothing and you audit the surface by reasoning — that is NOT "no findings". The safety tags they attach (FLUSH_NOT_BARRIER, POSTED_FLUSH_ONLY) are "confirm this", never a clearance.

The bug class (precise)

Cores coordinate across the NoC with NoC semaphores (L1 counters bumped by remote atomic increments / noc_semaphore_set_remote) and async NoC transactions whose completion is not implied by the issuing instruction returning. Misuse → data corruption (a consumer reads a remote write that hasn't landed, because the credit raced ahead of the data) or deadlock (a noc_semaphore_wait whose increment never arrives, or a multicast whose fan-out count is wrong). dataflow-cb-sync-audit covers the CB-credit abstraction (cb_*, RemoteCB); THIS audit covers the raw noc_semaphore_* + barrier handshakes those same kernels use directly, which the CB audit's grep does not reach.

Ground-truth mechanism (tt_metal/hw/inc/api/dataflow/dataflow_api.h, NoC docs)

• noc_async_read / noc_async_write return before the transfer completes. Completion is established only by noc_async_read_barrier / noc_async_write_barrier / noc_async_writes_flushed.
• noc_semaphore_wait(ptr, val) spins until the L1 semaphore reaches val. A REMOTE credit is noc_semaphore_inc (a remote NoC atomic) or noc_semaphore_set_remote / noc_semaphore_set_multicast (a remote 4-byte WRITE); bare noc_semaphore_set is a LOCAL store *sem=val (reset/init), NOT a remote signal — the tool deliberately excludes it (registry NOC_SIGNAL_CALLS). Multicast writes/incs fan out to N receivers.
• fence / memory-ordering caveats are NoC- and arch-specific; the barrier primitive must match the transfer type and NoC in use.

What to check

1. Data-before-signal ordering (the main race). When a producer writes a remote core's L1 and then signals readiness (a noc_semaphore_inc/set to that core), the data write must be flushed before the credit write — a noc_async_write_barrier / noc_async_writes_flushed between them. Otherwise the receiver's noc_semaphore_wait releases and it reads stale/partial L1. (CB analog of "flush before cb_push_back"; same discipline, raw form.) writes_flushed ≠ write_barrier: noc_async_writes_flushed only drains the initiator's outgoing queue (the write has left); noc_async_write_barrier waits for the remote acknowledgement (the write has landed). Same-VC transit-ordering (below) makes a mere flush sufficient when a later credit rides the same path — but a case that needs the data to have actually landed at the remote (cross-NoC, or an atomic credit) needs the ack barrier, not just a flush. A credit WRITE (set_remote/set_multicast/relay_* — never bare noc_semaphore_set, which is a LOCAL reset store) is the dominant false-positive class — do NOT flag "no flush" as a race by default. When the payload write and the credit write share initiator + destination + NoC + VC (multicast writes default to NOC_MULTICAST_WRITE_VC / static VC, or are coupled via NOC_CMD_VC_LINKED), same-VC packets are delivered in issue order, so the credit write cannot overtake the payload write — no flush needed. One exception, straight from Ordering.md: an L1 write followed by a write to an MMIO address has NO ordering guarantee — the MMIO write can race ahead. So a credit that targets a stream/config REGISTER rather than an L1 semaphore (the noc_inline_dw_write shape) is not in the safe class even same-NoC/VC/dest; treat it like an atomic credit and require the barrier. An ATOMIC credit (noc_semaphore_inc/inc_multicast/remote up) is NOT in that safe class by default. The data-before-credit race is that the increment is observed before the payload write is committed (landed) — and per the data-movement doc (tenstorrent/tt-low-level-documentation → data_movement_doc/general/posted_writes.md), noc_async_writes_flushed() guarantees only that the write departed the initiator, while noc_async_write_barrier() guarantees completion (ACK/landed). So do NOT assume a flush-only atomic is safe: whether same-VC issue-order actually commits the write before the atomic is a doc-grounded call (<arch>/NoC/Ordering.md + posted_writes.md), not an assumption — the tool flags it (FLUSH_NOT_BARRIER) precisely so you confirm rather than dismiss it. A barrier (completion) is likewise required for a credit that is an atomic multicast (inc_multicast), or for ANY credit that crosses separate command-buffer FIFOs (not order-preserving even same static VC on some archs), different NoCs (payload on default noc, flag on noc=1), or a different/unpinned VC. (A same-VC write multicast — set_multicast — is NOT in this list: per Ordering.md's same-VC route rule it is transit-ordered per-receiver like a unicast write, so it clears on a flush; only an atomic multicast needs the barrier.) A linked=true "fix" can itself hang (linked multicast + concurrent atomic on another cmd buffer). Ground every ordering call in <arch>/NoC/Ordering.md + the data-movement doc (the authoritative living sources) per race-audit-all's source ladder + Ground-or-abstain (missing living source ⇒ UNCERTAIN, never confirmed-RACE).
2. Semaphore balance & direction. Each noc_semaphore_wait(val) must be satisfiable by exactly the set of inc/set that reach it; an increment on a path whose wait is unconditional (or vice-versa) hangs or releases early. Producer waits for space / consumer waits for data — confirm the direction.
3. Multicast fan-out count. A noc_semaphore_wait(N) gated on a multicast must match the actual number of receivers that increment; an off-by-one in the grid/fan-out count → permanent stall.
4. Reuse-before-complete. A buffer reused (re-written, or freed) before its outstanding NoC transaction completes → corruption; confirm a barrier gates reuse.
5. Cross-kernel pairing. Producer (writer/reader on RISCV B/NC) and consumer live in different kernels — trace across the op's kernel set; a wait with no reachable remote incrementer is a deadlock, but only after chasing all participating kernels (a model/compute-layer kernel may issue the inc).

Additional kernel-tier NoC/L1 hazards (the read-side / exit / coherency checkers)

Same surface as checks 1–5, but distinct mechanisms — each has its own committed checker.

6. Inbound read consumed before its read-barrier (noc-read-barrier → READ_CONSUMED_BEFORE_BARRIER). noc_async_read* fills local L1 asynchronously; the data is present only after noc_async_read_barrier (which also invalidates the L1 cache on BH). If the buffer is consumed before that barrier — handed to compute via cb_push_back, or forwarded by tt_memmove (its 16B-aligned path is itself a NoC write whose SOURCE is the read buffer) — the consumer reads stale/partial L1. Correct idiom: noc_async_read(...); noc_async_read_barrier(); <consume>. Confirm the consumed buffer is actually the read's DESTINATION (the checker does no buffer-identity tracking — a push of an unrelated, already-drained buffer between the read and its barrier is a false candidate to refute; likewise a pipelined reader that prefetches buf_next while pushing buf_cur). Ground in <arch>/…/MemoryOrdering.md (read-request landing) + the noc_async_read_barrier semantics in dataflow_api.h. Fix = move the read-barrier before the first consumer.

7. Non-posted atomic left in flight at kernel exit (noc-atomic-exit → NO_ATOMIC_BARRIER_AT_EXIT). A remote atomic credit (noc_semaphore_inc / remote Semaphore::up) is tracked by a separate HW counter from writes (the non-posted-atomics counter); only noc_async_atomic_barrier() (or the all-draining noc_async_full_barrier()) drains it — a write barrier / writes_flushed does NOT. If kernel_main issues an atomic and returns without draining it, the atomic is in flight at exit: on Watcher/debug builds the post-kernel_main NOC-idle ASSERT can trip, and the receiver's readiness signal can be delayed past program teardown / noc_init across back-to-back launches (the release-build firmware epilogue does NOT drain in-flight atomics). Fix = noc.async_atomic_barrier() after the terminal .up(). Confirm it is genuinely the terminal atomic and no caller/callee drains it (the checker is scoped to kernel_main and can't see a helper's barrier). Ground in the non-posted-atomics counter contract (<arch> noc_nonblocking_api.h) + the brisck/ncrisck NOC-idle epilogue.

8. (Blackhole) hand-rolled L1 poll missing invalidate_l1_cache (noc-l1-invalidate → MISSING_L1_INVALIDATE, BH-only). On Blackhole the baby-RISC core caches L1 reads (write-through), so a NoC write into L1 does not invalidate the core's cached copy — a repeated plain *ptr poll of a remotely-written flag can return a stale cached value indefinitely (spin forever, or read stale route/counts). The framework's own noc_semaphore_wait calls invalidate_l1_cache() inside its spin (dataflow_api.h) — that is the ground-truth idiom a hand-rolled poll must replicate. Fix = use noc_semaphore_wait, or invalidate_l1_cache() each poll iteration. On WH there is no such read cache → the checker is BH-only. Severity is LATENT by default: the BH RISC L1 data cache ships disabled (configure_l1_data_cache(), tt_metal/hw/inc/internal/firmware_common.h) and is opt-in per RISC via TT_METAL_ENABLE_L1_DATA_CACHE_RISCVS=<BR,NC,TR*,ER*> — so file the finding as a real latent hang that reproduces only with the cache enabled, never as a live-today hang (and use that env var to force the repro). Confirm the BH L1 read-cache semantics against the BlackholeA0 memory/cache doc (ground-or-abstain: if the doc isn't reachable, the noc_semaphore_wait-invalidates-internally code fact still grounds the fix, but label the HW-cache claim needs-confirm rather than asserting it). The checker scopes to a get_semaphore-provenance poll or a function issuing NoC ops (a poll of a non-remote local flag needs no invalidate — refute those).

Method

1. Enumerate (scope reaches beyond tt-llk):

   # from the repo root (ttnn/ and models/ are siblings of tt_metal/, NOT under it)
   grep -rInE '\bnoc_semaphore_(wait(_min)?|set|inc|set_multicast(_loopback_src)?|inc_multicast|set_remote)\b|\bnoc_(async_write_barrier|async_writes_flushed|async_write_flushed_with_trid|async_write_barrier_with_trid|async_posted_writes_flushed|async_atomic_barrier|async_full_barrier|inline_dw_write)\b|\bnoc_async_read[a-z_]*\b|\bnoc_(fast_)?atomic_increment\b|\btt_memmove\b|\bcb_push_back(_hold_wr_ptr)?\b|\bremote_cb_push_back_and_write_pages\b|\.(up|set_multicast|set_multicast_loopback_src|relay_multicast|relay_unicast|inc_multicast|async_read|async_write_barrier|async_write_barrier_with_trid|async_writes_flushed|async_write_flushed_with_trid|async_read_barrier|async_read_barrier_with_trid|async_atomic_barrier|async_full_barrier|async_posted_writes_flushed)[[:space:]]*\(' \
     tt_metal/hw/inc/api ttnn/cpp models --include=*.h --include=*.cpp | grep -v '/tests/'
   

   Read-consumers (check 6). The grep lists the free-function CB-push forms (cb_push_back/cb_push_back_hold_wr_ptr/remote_cb_push_back_and_write_pages) and tt_memmove, mirroring the tool's is_read_consumer. The object-method form cb.push_back() is DELIBERATELY not grepped here (a bare .push_back( floods on std::vector; the tool gates it on the CB receiver type, which a grep can't) — it is enumerated by the dataflow-cb-sync grep, so pair the two for the method form.

   Both API forms. Modern ttnn kernels signal via the OBJECT form on a Semaphore — remote sem.up(noc, x, y, v) / sem.set_multicast(...) / relay_multicast / inc_multicast — and flush via the Noc object (noc.async_write_barrier()); the deterministic tool recalls these via registry.noc_op_of. The third alternation above catches them. Caveats: the LOCAL sem.up(value) / sem.set(value) (no NoC arg) are NOT remote signals — exclude them (the tool gates remote up on argc≥2); and .up( can match an unrelated receiver, so keep only Semaphore-typed receivers. Likewise the FREE-FUNCTION bare noc_semaphore_set(ptr,val) is a LOCAL store (reset/init), NOT a write credit — the tool EXCLUDES it (only noc_semaphore_set_remote / noc_semaphore_set_multicast[_loopback_src] / relay_* are write credits), so the grep's |set| over-matches it: exclude bare noc_semaphore_set from the data-before-signal analysis by hand. Exhaustive run — in scope, no sampling. The tt-llk header tree has ~0 NoC sites — that is expected (the primitives live one layer up) and is NOT a reason to report "no findings / out of layer." The sites are in tt_metal/hw/inc/api/dataflow + ttnn/cpp + models, which the grep above (and the frontmatter) target. Enumerate all matching kernels into the run's coverage ledger and fan out per kernel-family; do not sample.

2. Per cross-core handshake, pair the signaller (write + inc/set) with the waiter (noc_semaphore_wait + read). Confirm a flush/barrier sits between the data write and the credit, and between the read and any buffer reuse.

3. Run checks 1–5; for multicast, resolve the receiver count from the grid.

Verdict

• Data flushed before credit (ack barrier when the credit is atomic), balanced wait/inc with correct direction, multicast count matches, reuse gated by completion → SAFE.
• Credit reachable before the data write completes → DATA RACE — fix = order the write before the credit. A write credit (set_remote/multicast — not the local bare set) same-NoC/VC/dest is ordered by issue-order (a flush, or nothing, suffices — see check #1). An atomic credit (noc_semaphore_inc / remote up) needs the payload write committed (landed), which is noc_async_write_barrier (ACK); a bare noc_async_writes_flushed() guarantees only departure, not completion (data_movement_doc/general/posted_writes.md). Do NOT assume a flush-only atomic is safe on a same-VC unicast path — that it commits the write first is a doc call (<arch>/NoC/Ordering.md + posted_writes.md), so confirm it rather than dismiss it. A multicast / cross-command-buffer / cross-NoC-VC atomic credit definitely needs the barrier. (The noc-sync checker matches this: it clears an atomic inc only on a preceding barrier, and tags a flush-only inc safety=FLUSH_NOT_BARRIER — surfaced for you to confirm, not a pre-declared safe idiom.)
• Wait/inc imbalance or wrong fan-out count on a reachable path → DEADLOCK or early-release.
• Buffer reused before transaction completion → CORRUPTION — gate reuse with a barrier.
• Flush drains the wrong counter (noc_async_posted_writes_flushed for a NON-posted write/inc) → NO ORDERING (silent) — the posted and non-posted writes are tracked by two independent HW counters; noc_async_posted_writes_flushed() polls only the posted one, so it returns immediately (a no-op) when the credited write/inc was issued non-posted (the common default, e.g. a remote_cb write whose posted template arg is false). The correct departure guarantee for a non-posted write is noc_async_writes_flushed(). Confirm the write's posted-ness against the source before trusting the flush. (The noc-sync checker tags a credit whose only preceding flush is a posted-flush safety=POSTED_FLUSH_ONLY; note it fires only when the credit SIGNAL is visible in the kernel — a credit posted inside a primitive like remote_cb_push_back_and_write_pages is not surfaced and is yours to trace.)
• The credited "data" is a RISC L1 store, not a NoC write → a NoC flush/barrier CANNOT order it. When a producer core writes its result into L1 with a plain (volatile) store and then signals a consumer (a remote noc_semaphore_inc/.up(), or a mailbox push) that reads that L1, the ordering point is the Baby-RISC Load/Store Unit, not the NoC. Per <arch>/TensixTile/BabyRISCV/MemoryOrdering.md ("Cross-core signalling"): a store retires before its write-request lands, the RISCV fence is a no-op, and noc_async_write_barrier()/writes_flushed() drain NoC write-acks — a different subsystem — so none of them force the store to be processed into the L1 bank before the signal departs. The only correct idiom is load-back + consume: read the just-stored address(es) back and consume the loaded value (e.g. into a register the signal path depends on, or asm volatile("" :: "r"(x))) before issuing the signal — the same-address store→load pair is processed in order, so the load returning proves the store landed, and consuming it stalls the signal's emission until the load response is obtained. STALLWAIT does NOT substitute here — the doc scopes STALLWAIT to "store to Tensix GPR/Backend-Config → store to push a Tensix instruction" (the reconfig-stall case), not cross-core L1 signalling. Flaky-test signature (masked by the signal's NoC round-trip; loses under store-queue/bank contention). The noc-sync checker will still surface the signal as NOC_SIGNAL_NO_FLUSH when no flush precedes it, but do NOT "fix" it with a barrier, and do NOT treat a signal as safe merely because a NoC barrier precedes it — check whether the real producer is a RISC store.
• Inbound noc_async_read consumed before its noc_async_read_barrier (check 6) → CORRUPTION — stale/partial L1 handed downstream. Fix = read-barrier before the first consumer. Refute if the consumed buffer isn't the read's destination (unrelated already-drained buffer / pipelined prefetch).
• Non-posted atomic in flight at kernel_main return (check 7) → LATENT — Watcher NOC-idle ASSERT can trip; receiver readiness can slip past teardown/noc_init. Fix = noc.async_atomic_barrier() after the terminal .up(). (A write barrier / writes_flushed does NOT drain the atomic counter.)
• (Blackhole) hand-rolled L1 poll of a remote flag with no invalidate_l1_cache (check 8) → STALE-READ / HANG on BH — cached L1 read never refreshes; latent (the BH RISC L1 D-cache is disabled by default, opt-in via TT_METAL_ENABLE_L1_DATA_CACHE_RISCVS). Fix = noc_semaphore_wait or per-iteration invalidate_l1_cache(). WH: no-op (not a bug). Confirm BH cache semantics per ground-or-abstain.
• Risk only in a specific op's kernel set → LATENT/author-level — name the kernel.

Architecture note

The dataflow API is largely arch-agnostic, but the ordering primitives and NoC atomic semantics differ per NoC/arch — verify the barrier used matches the transfer type and that multicast/atomic behavior holds on the target arch (WH/BH/QSR). Don't transfer a verdict across arches without checking the NoC layer.

Output

For each cross-core handshake: file:line of signaller (write+inc) and waiter (wait+read), barrier present/missing between data and credit, semaphore balance + direction, multicast fan-out count check, reuse-after-completion check, verdict (SAFE / DATA-RACE / DEADLOCK / CORRUPTION / LATENT) + one-line fix. End with totals.