noc-sync-audit

name

noc-sync-audit

description

Audit cross-core NoC + L1 synchronization in dataflow kernels — noc_semaphore_wait/set/inc balance and direction, multicast fan-out counts, data-before-signal NoC ordering (noc_async_write_barrier / noc_async_writes_flushed before a remote credit), AND the read-side / exit / coherency hazards of the same surface: an inbound noc_async_read consumed before its read-barrier, a non-posted atomic left in flight at kernel exit, and (Blackhole) a hand-rolled L1 poll missing invalidate_l1_cache. The half of dataflow that dataflow-cb-sync-audit (CB credits) does not reach. Use after touching reader/writer kernels, noc_semaphore_*, noc_async_*_barrier, or any cross-core handshake not expressed as a CB.

/noc-sync-audit — cross-core NoC handshake correctness

Ground-truth precedence: the live ISA doc (tt-isa-docs MCP, fetched each run) outranks every rule, table, and example baked into this skill — treat those as dated illustrations. If the live ISA doc contradicts a baked rule here, do NOT silently proceed: surface the conflict to the user and ask whether the baked rule should be overwritten, discarded, or kept. Default to the ISA doc.

MANDATORY — before any verdict, read the shared grounding policy. The per-architecture source ladder (which docs to consult), the ground-or-abstain rule, and the Source preflight (list the sources you'll consult with their reachability + hierarchy, then PAUSE for the user) are defined once in race-audit-all → .claude/skills/race-audit-all/SKILL.md. Your FIRST action is to Read that file and follow its "Ground-truth source ladder", "Ground-or-abstain", and "Source preflight" sections — they are load-bearing: a verdict produced without them is ungrounded and MUST NOT be reported. If that file genuinely cannot be read, say so and abstain rather than proceed ungrounded. (If you were spawned by a race-audit-all sweep — your prompt already lists the confirmed sources — skip the Source preflight and do not pause; the orchestrator ran it once.)

Coverage — floor, not ceiling. The grep patterns and site lists in this skill are a seed, not an exhaustive enumeration. After running them, widen the search with full reasoning. The techniques here are illustrative examples, not the allowed set — use any approach your reasoning suggests, including ones not listed: e.g. semantic search (by behavior/effect, not just token), resolving macros / wrappers / typedefs / indirection the literal pattern can't match, following the call graph to callers and callees, and diffing the WH/BH/QSR variants to catch a site present in one arch and missing in another. If you can find a hazard, primitive, or site the encoded patterns don't cover — by any means — pursue and report it; do not clamp a stronger analysis to this list or to these techniques. State any residual coverage gaps explicitly (no silent caps).

Execution — parallel by default. When enumeration yields more than a few sites/files, fan out concurrent Agent calls by default (one per file/subsystem, a fresh context each), saturating the available concurrency (~10–16 at once); go inline only for a trivial set. The per-file fan-out described under Thoroughness is the default, not an exhaustive-only option. The cross-referencing/synthesis of results stays sequential (it must follow the per-unit findings). The heavyweight Workflow tool still requires explicit multi-agent opt-in — it is the opt-in exhaustive tier, not the default. Don't over-spawn a tiny diff.

Persisting results — single writer, incremental. Agents only return their findings; they never write a shared file (no concurrent-write clobbering). If findings are persisted to a file, the orchestrator/caller is the sole writer and appends each wave's returns as they arrive — incremental, never only-at-the-end — so an interrupt preserves every completed wave's findings.

Recall preflight — run the tool first (augmentor, not a verdict)

Before enumerating, run the deterministic llk-audit NoC checkers for a complete known-pattern worklist. This class is backed by four committed checkers — noc-sync (write→credit ordering), noc-atomic-exit, noc-read-barrier, noc-l1-invalidate:

# from the tt-metal repo root; do NOT cd into the tool dir (run.sh self-locates)
tt_metal/tt-llk/.claude/tools/llk-audit/run.sh <arch> --full-jit   # kernel tier runs all four

All four are empty over the tt-llk headers — their surface is JIT-compiled kernels (ttnn/models), so they only emit findings when fed a KERNEL fact base via --full-jit (the on-request capture; runbook in race-audit-all). Treat each checker's findings[] as a floor, not a ceiling — widen per its blind_spots (each names its own gaps: e.g. noc-read-barrier over-matches non-issuing read helpers and doesn't verify buffer identity; noc-atomic-exit is scoped to kernel_main). If the kernel tier isn't built, the checkers report nothing and you audit the surface by reasoning — that is NOT "no findings". The safety tags they attach (FLUSH_NOT_BARRIER, POSTED_FLUSH_ONLY) are "confirm this", never a clearance.

The bug class (precise)

Cores coordinate across the NoC with NoC semaphores (L1 counters bumped by remote atomic increments / noc_semaphore_set_remote) and async NoC transactions whose completion is not implied by the issuing instruction returning. Misuse → data corruption (a consumer reads a remote write that hasn't landed, because the credit raced ahead of the data) or deadlock (a noc_semaphore_wait whose increment never arrives, or a multicast whose fan-out count is wrong). dataflow-cb-sync-audit covers the CB-credit abstraction (cb_*, RemoteCB); THIS audit covers the raw noc_semaphore_* + barrier handshakes those same kernels use directly, which the CB audit's grep does not reach.

Ground-truth mechanism (tt_metal/hw/inc/api/dataflow/dataflow_api.h, NoC docs)

• noc_async_read / noc_async_write return before the transfer completes. Completion is established only by noc_async_read_barrier / noc_async_write_barrier / noc_async_writes_flushed.
• noc_semaphore_wait(ptr, val) spins until the L1 semaphore reaches val. A REMOTE credit is noc_semaphore_inc (a remote NoC atomic) or noc_semaphore_set_remote / noc_semaphore_set_multicast (a remote 4-byte WRITE); bare noc_semaphore_set is a LOCAL store *sem=val (reset/init), NOT a remote signal — the tool deliberately excludes it (registry NOC_SIGNAL_CALLS). Multicast writes/incs fan out to N receivers.
• fence / memory-ordering caveats are NoC- and arch-specific; the barrier primitive must match the transfer type and NoC in use.

What to check

1. Data-before-signal ordering (the main race). When a producer writes a remote core's L1 and then signals readiness (a noc_semaphore_inc/set to that core), the data write must be flushed before the credit write — a noc_async_write_barrier / noc_async_writes_flushed between them. Otherwise the receiver's noc_semaphore_wait releases and it reads stale/partial L1. (CB analog of "flush before cb_push_back"; same discipline, raw form.) writes_flushed ≠ write_barrier: noc_async_writes_flushed only drains the initiator's outgoing queue (the write has left); noc_async_write_barrier waits for the remote acknowledgement (the write has landed). Same-VC transit-ordering (below) makes a mere flush sufficient when a later credit rides the same path — but a case that needs the data to have actually landed at the remote (cross-NoC, or an atomic credit) needs the ack barrier, not just a flush. A credit WRITE (set_remote/set_multicast/relay_* — never bare noc_semaphore_set, which is a LOCAL reset store) is the dominant false-positive class — do NOT flag "no flush" as a race by default. When the payload write and the credit write share initiator + destination + NoC + VC (multicast writes default to NOC_MULTICAST_WRITE_VC / static VC, or are coupled via NOC_CMD_VC_LINKED), same-VC packets are delivered in issue order, so the credit write cannot overtake the payload write — no flush needed. One exception, straight from Ordering.md: an L1 write followed by a write to an MMIO address has NO ordering guarantee — the MMIO write can race ahead. So a credit that targets a stream/config REGISTER rather than an L1 semaphore (the noc_inline_dw_write shape) is not in the safe class even same-NoC/VC/dest; treat it like an atomic credit and require the barrier. An ATOMIC credit (noc_semaphore_inc/inc_multicast/remote up) is NOT in that safe class by default. The data-before-credit race is that the increment is observed before the payload write is committed (landed) — and per the data-movement doc (tenstorrent/tt-low-level-documentation → ), guarantees only that the write the initiator, while guarantees (ACK/landed). So do assume a flush-only atomic is safe: whether same-VC issue-order actually commits the write before the atomic is a doc-grounded call ( + ), not an assumption — the tool flags it () precisely so you confirm rather than dismiss it. A barrier (completion) is likewise required for a credit that is an (), or for ANY credit that crosses (not order-preserving even same static VC on some archs), (payload on default noc, flag on ), or a . (A same-VC multicast — — is NOT in this list: per 's same-VC route rule it is transit-ordered per-receiver like a unicast write, so it clears on a flush; only an multicast needs the barrier.) A "fix" can itself hang (linked multicast + concurrent atomic on another cmd buffer). Ground every ordering call in + the data-movement doc (the authoritative living sources) (missing living source ⇒ , never confirmed-RACE).

Additional kernel-tier NoC/L1 hazards (the read-side / exit / coherency checkers)

Same surface as checks 1–5, but distinct mechanisms — each has its own committed checker.

6. Inbound read consumed before its read-barrier (noc-read-barrier → READ_CONSUMED_BEFORE_BARRIER). noc_async_read* fills local L1 asynchronously; the data is present only after noc_async_read_barrier (which also invalidates the L1 cache on BH). If the buffer is consumed before that barrier — handed to compute via cb_push_back, or forwarded by tt_memmove (its 16B-aligned path is itself a NoC write whose SOURCE is the read buffer) — the consumer reads stale/partial L1. Correct idiom: noc_async_read(...); noc_async_read_barrier(); <consume>. Confirm the consumed buffer is actually the read's DESTINATION (the checker does no buffer-identity tracking — a push of an unrelated, already-drained buffer between the read and its barrier is a false candidate to refute; likewise a pipelined reader that prefetches buf_next while pushing buf_cur). Ground in <arch>/…/MemoryOrdering.md (read-request landing) + the noc_async_read_barrier semantics in dataflow_api.h. Fix = move the read-barrier before the first consumer.

7. Non-posted atomic left in flight at kernel exit (noc-atomic-exit → NO_ATOMIC_BARRIER_AT_EXIT). A remote atomic credit (noc_semaphore_inc / remote Semaphore::up) is tracked by a separate HW counter from writes (the non-posted-atomics counter); only noc_async_atomic_barrier() (or the all-draining noc_async_full_barrier()) drains it — a write barrier / writes_flushed does NOT. If kernel_main issues an atomic and returns without draining it, the atomic is in flight at exit: on Watcher/debug builds the post-kernel_main NOC-idle ASSERT can trip, and the receiver's readiness signal can be delayed past program teardown / noc_init across back-to-back launches (the release-build firmware epilogue does NOT drain in-flight atomics). Fix = noc.async_atomic_barrier() after the terminal .up(). Confirm it is genuinely the terminal atomic and no caller/callee drains it (the checker is scoped to kernel_main and can't see a helper's barrier). Ground in the non-posted-atomics counter contract (<arch> noc_nonblocking_api.h) + the / NOC-idle epilogue.

Method

1. Enumerate (scope reaches beyond tt-llk):

   # from the repo root (ttnn/ and models/ are siblings of tt_metal/, NOT under it)
   grep -rInE '\bnoc_semaphore_(wait(_min)?|set|inc|set_multicast(_loopback_src)?|inc_multicast|set_remote)\b|\bnoc_(async_write_barrier|async_writes_flushed|async_write_flushed_with_trid|async_write_barrier_with_trid|async_posted_writes_flushed|async_atomic_barrier|async_full_barrier|inline_dw_write)\b|\bnoc_async_read[a-z_]*\b|\bnoc_(fast_)?atomic_increment\b|\btt_memmove\b|\bcb_push_back(_hold_wr_ptr)?\b|\bremote_cb_push_back_and_write_pages\b|\.(up|set_multicast|set_multicast_loopback_src|relay_multicast|relay_unicast|inc_multicast|async_read|async_write_barrier|async_write_barrier_with_trid|async_writes_flushed|async_write_flushed_with_trid|async_read_barrier|async_read_barrier_with_trid|async_atomic_barrier|async_full_barrier|async_posted_writes_flushed)[[:space:]]*\(' \
     tt_metal/hw/inc/api ttnn/cpp models --include=*.h --include=*.cpp | grep -v '/tests/'
   

   Read-consumers (check 6). The grep lists the free-function CB-push forms (cb_push_back/cb_push_back_hold_wr_ptr/remote_cb_push_back_and_write_pages) and tt_memmove, mirroring the tool's is_read_consumer. The object-method form cb.push_back() is DELIBERATELY not grepped here (a bare .push_back( floods on std::vector; the tool gates it on the CB receiver type, which a grep can't) — it is enumerated by the dataflow-cb-sync grep, so pair the two for the method form.

   Both API forms. Modern ttnn kernels signal via the OBJECT form on a Semaphore — remote sem.up(noc, x, y, v) / sem.set_multicast(...) / relay_multicast / inc_multicast — and flush via the Noc object (noc.async_write_barrier()); the deterministic tool recalls these via registry.noc_op_of. The third alternation above catches them. Caveats: the LOCAL sem.up(value) / sem.set(value) (no NoC arg) are NOT remote signals — exclude them (the tool gates remote up on argc≥2); and .up( can match an unrelated receiver, so keep only Semaphore-typed receivers. Likewise the FREE-FUNCTION bare noc_semaphore_set(ptr,val) is a LOCAL store (reset/init), NOT a write credit — the tool EXCLUDES it (only / / are write credits), so the grep's over-matches it: exclude bare from the data-before-signal analysis by hand. The tt-llk header tree has ~0 NoC sites — that is (the primitives live one layer up) and is a reason to report "no findings / out of layer." The sites are in + + , which the grep above (and the frontmatter) target. Enumerate matching kernels into the run's coverage ledger and fan out per kernel-family; do not sample.

Verdict

• Data flushed before credit (ack barrier when the credit is atomic), balanced wait/inc with correct direction, multicast count matches, reuse gated by completion → SAFE.
• Credit reachable before the data write completes → DATA RACE — fix = order the write before the credit. A write credit (set_remote/multicast — not the local bare set) same-NoC/VC/dest is ordered by issue-order (a flush, or nothing, suffices — see check #1). An atomic credit (noc_semaphore_inc / remote up) needs the payload write committed (landed), which is noc_async_write_barrier (ACK); a bare noc_async_writes_flushed() guarantees only departure, not completion (data_movement_doc/general/posted_writes.md). Do NOT assume a flush-only atomic is safe on a same-VC unicast path — that it commits the write first is a doc call (<arch>/NoC/Ordering.md + posted_writes.md), so confirm it rather than dismiss it. A multicast / cross-command-buffer / cross-NoC-VC atomic credit definitely needs the barrier. (The noc-sync checker matches this: it clears an atomic inc only on a preceding barrier, and tags a flush-only inc safety=FLUSH_NOT_BARRIER — surfaced for you to confirm, not a pre-declared safe idiom.)
• Wait/inc imbalance or wrong fan-out count on a reachable path → DEADLOCK or early-release.
• Buffer reused before transaction completion → CORRUPTION — gate reuse with a barrier.
• Flush drains the wrong counter (noc_async_posted_writes_flushed for a NON-posted write/inc) → NO ORDERING (silent) — the posted and non-posted writes are tracked by two independent HW counters; noc_async_posted_writes_flushed() polls only the posted one, so it returns immediately (a no-op) when the credited write/inc was issued non-posted (the common default, e.g. a remote_cb write whose posted template arg is false). The correct departure guarantee for a non-posted write is noc_async_writes_flushed(). Confirm the write's posted-ness against the source before trusting the flush. (The noc-sync checker tags a credit whose only preceding flush is a posted-flush safety=POSTED_FLUSH_ONLY; note it fires only when the credit SIGNAL is visible in the kernel — a credit posted a primitive like is not surfaced and is yours to trace.)

Architecture note

The dataflow API is largely arch-agnostic, but the ordering primitives and NoC atomic semantics differ per NoC/arch — verify the barrier used matches the transfer type and that multicast/atomic behavior holds on the target arch (WH/BH/QSR). Don't transfer a verdict across arches without checking the NoC layer.

Output

For each cross-core handshake: file:line of signaller (write+inc) and waiter (wait+read), barrier present/missing between data and credit, semaphore balance + direction, multicast fan-out count check, reuse-after-completion check, verdict (SAFE / DATA-RACE / DEADLOCK / CORRUPTION / LATENT) + one-line fix. End with totals.