mirage-graph-coverage

# mirage megakernel graph-coverage auditor Catches the **missing-wiring** bug class in a mirage/MPK generated megakernel: a tensor (or whole feature) is defined and allocated but never wired into a task, so the kernel runs without it. The canonical case is GLM-5.1 on GB300 where the RoPE `cos`/`sin` tables were attached but consumed by **0 tasks**, so attention ran position-blind and the model emitted fluent-but-incoherent, repetition-collapsing tokens - while every per-kernel numerics test passed. ## Quick start ```bash python3 assets/mk-graph-coverage-audit.py <gendir> # <gendir> holds the generated kernel_<rank>.cu + task_graph_<rank>.json # (e.g. mirage demo/deepseek_v3/). Defaults to /work/mirage-perop2/demo/deepseek_v3 # options: --critical rope_cos,rope_sin,<scale-tensor> --ranks 0,1,2,3 ``` Exit 0 = PASS (every critical tensor consumed as input), 1 = FAIL (a critical tensor is declared but never consumed = missing wiring), 2 = no artifacts. ## What it does Per TP rank, it parses two generated artifacts and diffs them: - **declared** = `all_tensors["NAME"] = NAME;` lines in `kernel_<rank>.cu` - **consumed** = `base_ptr` of every `all_tasks[].inputs[]` / `.outputs[]` in `task_graph_<rank>.json`, mapped to the set of consuming `task_type`s + counts It then classifies each declared tensor and FAILs iff a *critical* tensor (`rope_cos`,`rope_sin`,`cos_pos_embed`,`sin_pos_embed` by default) has **0 input consumers**. ## Reading the output - `[OK] <t>: input_refs=N consumer_task_types=[...]` - wired (consumed by a task). - `[MISSING-WIRING] <t>: input_refs=0` - declared but never read by any task => the bug. - `INFO write-only` - produced by a task but never read (often a final output like `output_token`. Suspicious only for an intermediate/feature tensor). - `INFO dead` - declared, 0 refs at all (allocated scratch / codegen leftover. The `nullptr` sentinel shows here as a benign false positive). ## Worked example (GLM-5.1 on GB300) Before the wiring fix the auditor FAILs: `rope_cos` `input_refs=0` (rope absent from the graph). After the fix it PASSes: `rope_cos`/`rope_sin` `input_refs=156`, `consumer_task_types=[278, 296]` (296 = q-rope-apply, 278 = k_pe / KV-cache gather). Same artifacts a relL2 numerics harness would have called clean in both states - graph-coverage is what distinguishes them. ## Triage rule For a megakernel coherence bug (compiles, exits 0, incoherent output), run the graph-coverage audit **before** building a per-kernel numerics harness. A missing/mis-wired task makes a numerics harness pass while the model stays wrong, the audit names the unwired tensor directly. Only once coverage is clean does a relL2 harness make sense. ## Caveat Verifies **structural wiring** (is the tensor consumed by a task), not runtime numerical coherence. A PASS removes "feature absent" as the suspect. It does not by itself prove the output is correct.