mlir-air

Use when an NPU kernel passes its standalone shape test but produces NaN, garbage, or stale values when invoked as part of a larger pipeline. Common symptoms: correct first invocation but wrong on subsequent calls; correct in isolation but wrong when chained with other kernels.

Use when NPU FlashAttention hangs (`ERT_CMD_STATE_TIMEOUT`) or produces NaN at head_dim ≥ 128. Discriminates the three known root causes (compile-flag mismatch, seq-first dk_chunks bug, true L1 overflow) via a symptom-classification table and applies the documented fix.

Use when stitching kernels into a multi-launch ELF and the AIE compiler rejects the merged module (BD exhaustion, channel routing, herd shape conflict, IR validation error, DMA stride limitation). Discriminates the 6 known compile blockers via a symptom-classification table.

Entry point for deploying a new decoder-only LLM on AMD NPU2. Invoked by the user as `/deploy-new-llm <hf_model_id> [--name <dirname>] [--target npu2|npu1] [--dtype bf16|fp16]`. Bootstraps the per-model workspace, validates architecture is in scope, and dispatches the 7 per-phase skills with the gate of each phase enforced by that phase's skill.

Optimization skill — reuse NPU BufferObjects across calls instead of re-allocating/re-writing them. Two mechanics in one class: (B1) per-layer weight BOs pre-loaded once and skipped via static_input_indices, and (B2) intermediate BOs the kernel overwrites, skipped via intermediate_indices. Invoked by phase-4-prefill-optimization and phase-5-decode-optimization to cut redundant host↔NPU data movement. Decode amplifies the weight-BO win (weights reused on every token).

Optimization skill — choose activation layouts so consecutive kernels hand off on-device without a host-side transpose. Canonical case: seq-first (seq, n_heads·head_dim) so RMSNorm → RoPE → FlashAttention → O-proj stay seq-first, eliminating 1–4 host transposes per layer. Invoked by phase-4-prefill-optimization (and phase-5 when decode introduces a transpose phase-4 didn't fix).

Procedural recipe for fusing multiple `air.launch` kernels into one multi-launch ELF (single XRT invocation). Invoked by phase-4-prefill-optimization and phase-5-decode-optimization to fuse kernel groups when building NEW model-specific fused ELFs (kernel-first path). Reduces XRT dispatch overhead (~50–200 µs per call on NPU2).

Phase 0 of LLM deployment — produce `<model>_weights.py` (HF weight loader) and `<model>_cpu_helpers.py` (the few NumPy helpers production prefill/decode import), then confirm the HF bf16 reference baseline loads and runs via the shared `programming_examples/llms/verify/` subsystem's HfRunner. Downstream phases compare NPU against HF transformers in bf16 directly; there is no hand-written full-model FP32 oracle.

Phase 1 of LLM deployment — for every leaf kernel × shape the model needs, verify numerical correctness on real NPU2 against the registry's GPU/vLLM-aligned standard. Primary gate where a standalone harness exists: the harness's full-output element-wise `np.isclose` check at that kernel's `rtol`/`atol` vs an FP32 reference (the same PASS/FAIL `make run` prints). Fallback for a kernel with no harness: `make diagnosis` per-layer cosine vs the HF bf16 reference. Record each verified (kernel, shape) as a row in that kernel's "tested shapes" table in `kernel_registry/supported_kernels.md` + `details/<Kernel>_bf16.md` (Used by = `<model>`); per-model progress stays in `<model>/docs/`. Hard gate before integration. Invoked by deploy-new-llm after Phase 0.

Phase 2 of LLM deployment — wire the verified Phase 1 kernels into one transformer block on NPU and verify per-layer cosine vs the HF bf16 reference (the shared `programming_examples/llms/verify/` diagnosis lens, promoted to a gate at layer 0). Catches integration bugs (layout mismatches, missing transposes, type drops between kernel boundaries) before scaling to N layers.

Phase 3 of LLM deployment — wire all N layers and verify NPU matches the HF bf16 reference end-to-end (per-layer cosine via the shared `programming_examples/llms/verify/` diagnosis lens + token-level top-5 set-inclusion via its token-set gate) at canonical prompts. Catches accumulated drift, KV cache bugs, layer-indexed weight loading errors. Invoked after Phase 2 gate.

Phase 4 of LLM deployment — apply the shared optimization skillset to a Phase-3-correct prefill pipeline (multi-launch merge, BO pre-loading + intermediate buffer reuse, seq-first layout). Thin orchestrator that dispatches `opt-merge-multi-launch-kernels`, `opt-buffer-object-reuse`, and `opt-layout-alignment`. Each step preserves correctness by re-running the Phase 3 gate — `make verify` (token-set vs HF bf16) is the PASS/FAIL gate; `make diagnosis` per-layer cosine is the informational lens used to localize a regression. Invoked after Phase 3 PASS.

Phase 5 of LLM deployment — apply the shared optimization skillset to a Phase-4-correct decode pipeline (multi-launch merge with N-way extern rename, static weight BOs, on-device layout). Thin orchestrator that dispatches `opt-merge-multi-launch-kernels`, `opt-buffer-object-reuse`, and `opt-layout-alignment`. Each step preserves correctness by re-running the Phase 3 gate — `make verify` (token-set vs HF bf16) is the PASS/FAIL gate; `make diagnosis` per-layer cosine is the informational lens used to localize a regression. Invoked after Phase 4 PASS.

Phase 6 of LLM deployment — integrate Phase 4 prefill + Phase 5 decode into a clean `<model>_inference.py`, write the model's `verify_adapter.py` hooking into the shared `programming_examples/llms/verify/` subsystem + a Makefile (run / verify / verify-full / diagnosis / profile), and confirm `make verify` (top-k token-set gate vs HF bf16) PASSES. That gate is the production-readiness check. Capture lessons learned. Invoked after Phase 5 PASS.

Phase 7 of LLM deployment — spawn a fresh subagent that treats the deployment as UNTRUSTED, audits the `make verify` implementation (anti-reward-hacking: confirms the token-set gate runs the production path vs HF bf16), then re-runs it as the primary gate. Produces a structured `evaluation_report.md` a human can read in 2 minutes to know the full deployment state. Invoke as `/phase-7-independent-evaluator <model_dir>` or auto-spawn from deploy-new-llm after Phase 6 PASS.