Cfg Word Overlap Audit

/cfg-word-overlap-audit — Cross-thread CFG register-file word overlap

Ground-truth precedence: the live ISA doc (tt-isa-docs MCP, fetched each run) outranks every rule, table, and example baked into this skill — treat those as dated illustrations. If the live ISA doc contradicts a baked rule here, do NOT silently proceed: surface the conflict to the user and ask whether the baked rule should be overwritten, discarded, or kept. Default to the ISA doc.

MANDATORY — before any verdict, read the shared grounding policy. The per-architecture source ladder (which docs to consult), the ground-or-abstain rule, and the Source preflight (list the sources you'll consult with their reachability + hierarchy, then PAUSE for the user) are defined once in race-audit-all → .claude/skills/race-audit-all/SKILL.md. Your FIRST action is to Read that file and follow its "Ground-truth source ladder", "Ground-or-abstain", and "Source preflight" sections — they are load-bearing: a verdict produced without them is ungrounded and MUST NOT be reported. If that file genuinely cannot be read, say so and abstain rather than proceed ungrounded. (If you were spawned by a race-audit-all sweep — your prompt already lists the confirmed sources — skip the Source preflight and do not pause; the orchestrator ran it once.)

Coverage — floor, not ceiling. The grep patterns and site lists in this skill are a seed, not an exhaustive enumeration. After running them, widen the search with full reasoning. The techniques here are illustrative examples, not the allowed set — use any approach your reasoning suggests, including ones not listed: e.g. semantic search (by behavior/effect, not just token), resolving macros / wrappers / typedefs / indirection the literal pattern can't match, following the call graph to callers and callees, and diffing the WH/BH/QSR variants to catch a site present in one arch and missing in another. If you can find a hazard, primitive, or site the encoded patterns don't cover — by any means — pursue and report it; do not clamp a stronger analysis to this list or to these techniques. State any residual coverage gaps explicitly (no silent caps).

Execution — parallel by default. When enumeration yields more than a few sites/files, fan out concurrent Agent calls by default (one per file/subsystem, a fresh context each), saturating the available concurrency (~10–16 at once); go inline only for a trivial set. The per-file fan-out described under Thoroughness is the default, not an exhaustive-only option. The cross-referencing/synthesis of results stays sequential (it must follow the per-unit findings). The heavyweight Workflow tool still requires explicit multi-agent opt-in — it is the opt-in exhaustive tier, not the default. Don't over-spawn a tiny diff.

Persisting results — single writer, incremental. Agents only return their findings; they never write a shared file (no concurrent-write clobbering). If findings are persisted to a file, the orchestrator/caller is the sole writer and appends each wave's returns as they arrive — incremental, never only-at-the-end — so an interrupt preserves every completed wave's findings.

Recall preflight — run the llk-audit tool first (augmentor, not a verdict)

Get the deterministic candidate list before manual analysis (it resolves every cfg write to its 32-bit word via cfg_defines.h, per register file, and attributes the thread):

tt_metal/tt-llk/.claude/tools/llk-audit/run.sh <wormhole|blackhole|quasar> --checks cfg-word-overlap
# from the tt-metal repo root; do NOT cd into the tool dir (run.sh self-locates)
# PR-scoped: add --changed [BASE] (default main) to report only words touching a changed file.
# candidates: out/audit.<arch>.json -> .checks["cfg-word-overlap"].findings

CROSS_THREAD_SHARED_WORD = a 32-bit word ≥2 threads write — a candidate, not a race. The tool PRE-COMPUTES the bit-disjoint-masking check into the finding's safety field (SAFE_BY_MASKING = all cross-thread writers are byte-atomic RMW on disjoint bits; POTENTIAL_CLOBBER = a full-word/non-atomic/overlapping writer; UNKNOWN = a mask didn't resolve; UNRESOLVED_COWRITER = fewer than 2 known threads (a lone known thread OR all-unknown writers) + an unattributable co-writer, a low-confidence widen) — READ .findings[].safety rather than redoing the mask analysis; you still verify semaphore/mutex ordering and value-invariance. UNRESOLVED = a field that didn't resolve to an ADDR32 (resolve it by hand). INTRA_THREAD_CLOBBER = a full-word write to a multi-field word where the same thread masked-writes a sibling field elsewhere (pattern 3) — a candidate (an intentional whole-word set is benign). The tool separates Config vs ThreadConfig by the write instruction (SETC16 → ThreadConfig; every other write → Config; THCON is a sub-range of Config, not its own file) — widen for the multi-bank SETC16-aliasing nuances below and any field the ADDR32 regex missed. The tool never clears a word; you decide. If unbuilt, proceed manually.

The bug class (precise)

The three Tensix threads (T0=unpack, T1=math, T2=pack) do not share GPR files, but they all write the shared backend Config register file (Config[2][...] at TENSIX_CFG_BASE). LLK addresses it by named field (<REG>_ADDR32 word index + _MASK/_SHAMT). Two differently-named fields can occupy the same 32-bit word — invisible from the names alone, visible only when you resolve _ADDR32 to a number. If different threads write the same word, you can lose one thread's field. The classic example: STACC_RELU_* (packer) and ALU_ACC_CTRL_Zero_Flag_disabled_* (math/unpack) share a word; a packer full-word write zeroes the math field.

HW rules that decide whether an overlap is a RACE (ground these first)

From the tt-isa-docs (RMWCIB.md, BackendConfiguration.md — fetch via the tt-isa-docs MCP for WH/BH):

• One Configuration Unit serializes all config writes → no bit-tearing; every WRCFG/RMWCIB/REG2FLOP/RISC sw completes atomically w.r.t. other config writes.
• RMWCIB is atomic per byte: *byte = (new & mask) | (old & ~mask). Two different threads RMW'ing the same word with disjoint masks is therefore always safe, regardless of order.
• Config has 2 banks selected by the issuing thread's CFG_STATE_ID ("any thread can access any bank"). In steady state all three threads run on bank 0 (cfg_state_id==0), so low words ARE physically shared. flip_cfg_state_id/TTI_SETC16(CFG_STATE_ID_StateID,…) only diverges briefly (e.g. llk_math_fast_tilize). Words >= GLOBAL_CFGREG_BASE_ADDR32 are single-copy (write hits both banks).

The patterns that can corrupt

1. Full-word write across a register boundary (cross-thread) — TTI_WRCFG(..., p_cfg::WRCFG_32b, ADDR) or RISC cfg[ADDR]=/sw — writes all 32 bits, so it overwrites every other field in that word owned by another thread. This is the sharp bug. MOP/replay-embedded form: a TT_OP_WRCFG(..., WRCFG_32b, ADDR) baked into a ckernel_template/replay program (e.g. llk_unpack_untilize.h) is the SAME full-word write, executed when the MOP runs — the deterministic tool EXCLUDES TT_OP_* (an opcode VALUE, not a directly-issued instruction, so its runtime target isn't statically known), so these are yours to assess by MOP semantics; the grep seeds them but confirm the MOP's target word before flagging.
2. Two threads RMW the SAME field bits with possibly-different values → last-writer-wins. Correct value then depends entirely on inter-thread ordering.
3. Intra-thread full-word clobber (no second thread needed) — a full-word write (WRCFG_32b/cfg[ADDR]=) to a word that the same thread also populates elsewhere via a masked write: the full-word write builds its value from only its own field and writes 0 into the sibling field the thread set earlier. A deterministic destructive overwrite, not a concurrency race — so a mutex/semaphore cannot fix it; the fix is still a masked RMW. This is exactly the pattern the "≥2 distinct threads" reduction in step 2 silently drops — hunt it with query (b) below. Example: Quasar THCON_PACKER1_REG0 SRC_ADDR_OFFSET full-word write zeroing the INSTRN_LOOP_COUNT/INSTRN_COUNT auto-loop bits set elsewhere by the same packer thread.

Safe by construction (do NOT flag)

• Disjoint-mask cfg_reg_rmw_tensix<> / RMWCIB from different threads to the same word (byte-atomic).
• A field written by only one thread.
• t6_mutex_acquire(mutex::REG_RMW) — but only protects parties that take it. ⚠️ The math thread never takes REG_RMW; unpack/pack take it in some paths only. A mutex'd unpack write is NOT protected against a concurrent math RMW of the same word — that case falls back to rules 1/2 + dataflow sync.

Method

1. Build the word→field map from cfg_defines.h (per arch) and cross-reference thread accesses. Threads by file: llk_unpack*/cunpack_common=UNPACK, llk_math*/cmath_common=MATH, llk_pack*/cpack_common=PACK. CFG defines:

   • WH: tt_metal/hw/inc/internal/tt-1xx/wormhole/wormhole_b0_defines/cfg_defines.h
   • BH: tt_metal/hw/inc/internal/tt-1xx/blackhole/cfg_defines.h
   • Quasar: tt_metal/hw/inc/internal/tt-2xx/quasar/cfg_defines.h Parse #define <X>_ADDR32 <n>, <X>_MASK, <X>_SHAMT; resolve *_RMW macros (first token is the *_ADDR32). Enumerate EVERY config-write mechanism — not just the masked-RMW/WRCFG pair. A "no other writer" / "SAFE by disjoint masks" conclusion is itself a negative, so it is only valid if the write-mechanism sweep is complete — a writer you didn't map is a race you didn't see (ground-or-abstain applies to this negative exactly as race-audit-all states it). Map every access:
   • cfg[SYM]= / RISC sw →WR_FULL (mask 0xffffffff); =cfg[SYM]→RD.
   • cfg16[SYM]= (via get_cfg16_pointer())→a 16-bit HALF-WORD write — a MASKED co-writer of the 32-bit word (NOT a full-word clobberer), so a bare cfg[ grep misses it (the 16 breaks the literal).
   • cfg_reg_rmw_tensix<SYM…> / TTI?_RMWCIB*(...,SYM)→byte-atomic RMW(field mask).
   • cfg_rmw(SYM_RMW,...) / cfg_rmw_gpr(...)→software (RISC read-modify-write) RMW — NOT byte-atomic, so disjoint-mask does NOT make it cross-thread-safe (needs mutex::REG_RMW); resolve SYM_RMW→SYM_ADDR32.
   • TTI_WRCFG(...,WRCFG_32b,SYM)→full word (WRCFG_128b→four words [SYM..SYM+3]); also TT_OP_WRCFG(...) baked into a MOP/replay buffer, which a plain TTI_WRCFG grep misses.
   • TTI?_REG2FLOP(size,...,SYM - THCON_CFGREG_BASE_ADDR32, gpr)→sized (4B/2B/1B) GPR→config-flop write (target is flop-relative; resolve by adding THCON_CFGREG_BASE_ADDR32 back).
   • TTI?_CFGSHIFTMASK(...,SYM)→config shift-mask write.
   • TTI_SETC16(SYM, val16)→16-bit set — but SETC16 targets a physically DIFFERENT register file (see the SETC16-aliasing caveat below); do not merge its _ADDR32 into the ALU/THCON word map by the number alone.

   A scripted parser is the reliable way (resolve names→numeric addr; group by addr; flag addresses written by ≥2 threads). Quick grep to spot candidates by hand:

   cd tt_metal/tt-llk
   grep -rInE "cfg_reg_rmw_tensix<|\bcfg_rmw(_gpr)?\(|TTI?_WRCFG\(|TT_OP_WRCFG\(|TTI?_RMWCIB|TTI?_REG2FLOP|TTI?_CFGSHIFTMASK|TTI?_SETC16\(|get_cfg16_pointer\(|cfg(16)?\[[A-Za-z_]" tt_llk_* --include=*.h | grep -v /tests/
   

   SETC16 aliasing — do NOT infer a race from a bare index collision. TTI_SETC16 writes the THREAD/local-control cfgreg file (state-id, address-mods, SRCA_SET, CLR_DVALID, FP16A_FORCE, UNPACK_MISC_CFG, …) — a different physical register file whose _ADDR32 numbers alias the low ALU/THCON words because two files are both based at 0 — the ALU/THCON cfg file (ALU_CFGREG_BASE_ADDR32) and, on arches that have one, the THREAD/local-control file (THREAD_CFGREG_BASE_ADDR32). Derive every bank base from the target arch's cfg_defines.h at audit time (grep '#define .*_CFGREG_BASE_ADDR32') — never from baked numbers: both the values and the set of banks are arch-specific. (Illustrative only — re-derive, do not trust: WH ALU=THREAD=0, PACK0=8, UNPACK0=36, UNPACK1=44, THCON=52, GLOBAL=152; BH shifts these to PACK0=12, UNPACK0=44, UNPACK1=56, THCON=64, GLOBAL=180; Quasar exposes a different, smaller set — only ALU=0, THCON=12, GLOBAL=80, with no separate PACK/UNPACK/THREAD banks.) A numeric ADDR32 collision between a SETC16 target and an ALU/THCON field is an index alias, not the same physical word. The tell: CFG_STATE_ID_StateID sits at word 0, bit 0 — the same word-0/bit-0 slot that ALU_FORMAT_SPEC_REG_SrcA_val (bits 0–3) also occupies — yet set_cfg_state_id rewrites it on every state flip without corrupting the ALU format. Ground any such collision in the *_CFGREG_BASE_ADDR32 bank layout before deciding; never flag it from the index alone.

2. Run BOTH reductions — the second catches the intra-thread clobber the first structurally misses:

   • (a) Cross-thread: keep words written by ≥2 distinct threads; OR together each thread's write-mask and compute pairwise bit-overlap.
   • (b) Full-word-write vs multi-field word (ANY thread count): keep every word that receives a full-word write (WRCFG_32b / cfg[ADDR]= / RISC sw) AND holds ≥2 distinct named fields — regardless of how many threads touch it. The full-word write clobbers every sibling field in that word; a single-thread instance is still a real destructive overwrite (pattern 3). Do not drop a word just because only one thread writes it.

3. Classify with the rules above:

   • any thread does a full-word write to the word → inspect: does it cross into another thread's field bits? (pattern 1)
   • two threads' write-masks share bits → same-field race (pattern 2)
   • all masks disjoint → SAFE (byte-atomic), note as fragile.

4. For each non-safe word, check ordering: is there a mutex::REG_RMW taken by all writers (rare — math doesn't)? Otherwise the only thing preventing concurrency is the dataflow handshake (UNPACK_TO_DEST, MATH_PACK, MATH_DONE semaphores). Decide whether a reconfig on one thread can land while another thread's op (same StateID) reads the word.

Architecture difference — don't false-positive on Quasar

Quasar is cross-thread-clean on the ALU config by per-engine register namespacing + single-thread (MATH) ownership — not purely a "byte-per-operand" layout. The alu_config_t (tt_llk_quasar/common/inc/cmath_common.h) word 0 (ADDR32=0) IS byte-per-operand (SrcA_val:8 / SrcB_val:8 / Dstacc_val:8, each + an override bit), so those writes land in disjoint bytes — but words 1 and 2 still pack multiple fields (word 1 = ALU_ROUNDING_MODE + REG0_SrcA + REG1_SrcB; word 2 = REG2_Dstacc + ALU_ACC_CTRL_Fp32/SFPU_Fp32/INT8_math), exactly like WH/BH. The reason no cross-thread overlap arises is that all these ALU words are written only by the MATH thread (each engine has its own register namespace on Quasar), and relu lives in per-packer THCON_PACKER{0,1}_REG3_* (not packed with ALU_ACC_CTRL). Combined with Quasar HW write-ordering (AutoTTSync; read Confluence 1340276980 at audit for its actual ordering guarantees), the WH/BH cross-thread packed-word races don't arise. Verify the address map and thread-ownership still resolve for Quasar before concluding "none" (don't mistake a parser miss for absence) — and still run query (b): the intra-thread full-word clobber (pattern 3) DOES occur on Quasar even where cross-thread overlap doesn't (see Reference findings).

Verdict

• All cross-thread writers use disjoint masked RMWCIB → SAFE (byte-atomic). Note it's fragile: a future full-word write to that word breaks it.
• Two threads RMW the same field bits, ordered only by dataflow sync → LATENT/sync-dependent — works today, document the invariant; a misplaced reconfig races.
• A full-word write (WRCFG_32b/cfg[]=) overwrites another thread's field in the same word → RACE (real). Fix = make it a masked cfg_reg_rmw_tensix<…_MASK> touching only its own bits, under mutex::REG_RMW if a sibling path already does. (Cross-check: is the same register written safely elsewhere? An inconsistency between two writers of one register is the tell.)

Thoroughness (optional, full sweep)

The word→field cross-reference is best done by a deterministic script (parse cfg_defines → resolve every access → group by addr), not an agent. Then, only if the user opts into multi-agent orchestration, run a Workflow to adversarially verify each flagged word: confirm the bit layout from cfg_defines.h, confirm each writer's thread + mask + mutex/sync context, and try to refute "SAFE by disjoint masks" (look for any full-word writer to that word). Report live RACE vs latent/sync-dependent vs SAFE-but-fragile separately; never upgrade "probably ordered by sync" to SAFE without naming the semaphore.

Reference findings (ground truth — WH/BH audit)

• RACE — RESOLVED (canonical full-word-clobber example): _llk_pack_relu_config_ (llk_lib/llk_pack_common.h) writes the word holding ALU_ACC_CTRL_Zero_Flag_disabled_src/dst (bits 0-1, written by MATH cmath_common.h + UNPACK reduce paths). It used to full-word-clobber those bits via an unmasked TTI_WRCFG(..., WRCFG_32b, STACC_RELU_ApplyRelu_ADDR32); it is now a masked cfg_reg_rmw_tensix<STACC_RELU_ApplyRelu_ADDR32, 0, hw_relu_mask>(...) — the pattern-3 fix (see Safe by construction), mirroring configure_pack (cpack_common.h). Verify the masked form is still present before flagging — do not re-report the current code as a live RACE.
• LATENT / dead-writer (arch-specific — re-derive per arch, do NOT treat this line as static): the ALU word holding ALU_FORMAT_SPEC_REG0_SrcA / REG1_SrcB / ALU_ACC_CTRL_INT8_math_enabled is RMW'd by MATH (llk_math_common.h, llk_math_eltwise_unary_datacopy.h, transpose_dest, reduce — none under the mutex), and for SrcA also by UNPACK configure_unpack_AB (cunpack_common.h, under mutex). The UNPACK-side writers of INT8 (enable_int8_fpu_math / _llk_enable_int8_fpu_math_) and SrcB (_llk_unpack_reduce_init_) are dead code (zero callers) — on WH they are removed, so INT8/SrcB are MATH-only; on BH they still exist but are unreachable ⇒ LATENT-dead-writer (safe only because the UNPACK writer never runs). Genuine live UNPACK+MATH same-bit sharing (e.g. WH SrcA) is last-writer-wins, safe only because the pipeline semaphores keep format reconfig from overlapping the other thread's op. Confirm caller-liveness and per-arch presence before flagging — never inherit this entry as ground truth.
• SAFE-fragile: word holding ALU_FORMAT_SPEC_REG2_Dstacc (PACK, 25-28) + ALU_ACC_CTRL_Fp32/SFPU_Fp32 (MATH, 29-30) + INT8 (31); and BH THCON_SEC0_REG1 (UNPACK Unp_LF8_4b_exp bit22 vs PACK Pack_L1_Acc/Exp_threshold/Pac_LF8_4b_exp) — all disjoint masked RMWCIB → safe by byte-atomicity.
• Quasar (cross-thread): the ALU config words show no cross-thread overlap — not because the layout is fully byte-per-operand (words 1/2 still pack multiple fields), but because they are MATH-thread-owned (per-engine register namespacing) + AutoTTSync ordering. But the tool DOES surface 1 CROSS_THREAD_SHARED_WORD / UNRESOLVED_COWRITER: RISC_DEST_ACCESS_CTRL (word 3, ckernel_dest.h), where the template set_dest_fmt<t> writes SEC0/SEC1/SEC2_fmt via masked RMWCIB and the tool cannot statically resolve t (the thread). It is safe by disjoint masked SEC fields (each thread's SEC byte is distinct) — verify those masks are disjoint before dismissing, but it is not a confirmed race.
• Quasar (intra-thread clobber — pattern 3, found only by query (b)): _set_packer_dest_registers_<PACK1> (common/inc/ckernel_trisc_common.h) does an unmasked full-word cfg[THCON_PACKER1_REG0_SRC_ADDR_OFFSET_ADDR32]=… that zeroes INSTRN_LOOP_COUNT/INSTRN_COUNT set by _llk_pack_srcs_config_ (llk_lib/llk_srcs.h) on the same packer thread; per-tile in SyncHalf, latent until the SrcS→Packer1 path is wired (Packer0's word carries no INSTRN bits, so PACK0 is harmless). The cross-thread-only reduction reports "QSR clean" and misses this. Fix = masked cfg_rmw.

Output

For each shared word: numeric ADDR32, the fields in it + which thread writes each (file:line, primitive, mask), pairwise bit-overlap, mutex/sync context, arch, verdict (SAFE-fragile / LATENT-sync-dependent / RACE) with one-line reasoning, and the one-line fix for any RACE. End with totals per verdict per arch.