phase-3-full-model-validation

name	phase-3-full-model-validation
description	Phase 3 of LLM deployment — wire all N layers and verify NPU matches the HF bf16 reference end-to-end (per-layer cosine via the shared `programming_examples/llms/verify/` diagnosis lens + token-level top-5 set-inclusion via its token-set gate) at canonical prompts. Catches accumulated drift, KV cache bugs, layer-indexed weight loading errors. Invoked after Phase 2 gate.

Purpose

Phase 2 verified one transformer block. Phase 3 scales to all N layers and confirms the full prefill stays numerically aligned with the HF bf16 reference end-to-end. Catches: accumulated BF16 drift across deep stacks, KV cache layout bugs, layer-indexed weight loading errors, LM head precision drops.

Phase 3 reuses the verify/ subsystem's two lenses:

• make diagnosis — per-layer ffn_out cosine (NPU vs HF bf16) for ALL layers. Informational by default; Phase 3 promotes it to a gate.
• make verify — token-level top-5 set-inclusion (NPU vs HF bf16 greedy sequences). Already a hard gate (exit 0/1).

There is no hand-written CPU full-model forward — the reference is HF transformers bf16 throughout.

Phase 3 PASS criteria (HARD GATES)

Run make diagnosis (per-layer cosine, all layers) and make verify (token-set gate) on the canonical prompts. All must hold:

Semantic correctness (token-level, vs HF bf16 — THE GATE)

1. make verify PASSES: at the first divergence between NPU and HF greedy sequences, NPU's chosen token is in HF's top-5 AND HF's chosen token is in NPU's top-5 (the compute_topk_set_check gate, GATE_K=5, GATE_N_TOKENS=32), measured against HF bf16 on generated tokens. bf16 noise can flip top-1 even between mathematically equivalent implementations, but almost never displaces a token out of the top-5. This top-k token-set gate mirrors vLLM's check_logprobs_close — the GPU/industry-standard end-to-end signal, and the same gate Phase 6/7 re-run. This is the binding correctness verdict for Phase 3.

Numerical sanity (per-layer, vs HF bf16 — diagnostic, NOT the gate)

2. Per-layer cosine table recorded, eyeballed for gross failure. This is bf16-vs-bf16 (no FP32 ground truth at the layer level), and per-layer activation cosine is not an industry-standard correctness gate (see the note below) — so it does not PASS/FAIL the phase. Use it to localize a problem when criterion 1 fails: a layer where cosine collapses (e.g. < 0.85, or a sudden cliff |cos[i+1] − cos[i]| > 0.05) points at a layer-indexed bug (wrong bo_key=f"kernel_L{i}", weight-load shifted by a layer). If make verify PASSES, a gently drifting cosine (a deep stack reaching ~0.88 at the last layer) is expected bf16 geometry, not a bug.
3. Record max_abs / max_rel alongside the cosines (the diagnosis error_metrics reports them) as a regression baseline for future deployments at the same shape.

Hygiene

4. No NaN anywhere in the stack.
5. Per-layer cos table (from diagnosis) + make verify PASS/FAIL for every canonical prompt documented in <model>/docs/development_progress/phase3_full.md.

Why this split. A 2026 survey of industry practice (vLLM, HF transformers, llama.cpp, MLPerf, TensorRT-LLM, GPTQ/AWQ/SmoothQuant) found that correctness is gated end-to-end (vLLM token-set, llama.cpp logit-KL, MLPerf ≥99%-of-reference) — per-layer activation cosine is used for localization, not as a pass/fail gate, and where it is gated at all (TPU-MLIR) the bar is 0.99, not a loose value. Hence the token-set check is THE gate and cosine is the lens. A future upgrade could add a logit KL-divergence / perplexity-≥99%-of-HF check (llama.cpp / MLPerf style) as an even stronger end-to-end signal.

Knowledge base references

• programming_examples/llms/llama32_1b/llama32_1b_inference.py:run_npu_prefill — reference full-stack pipeline (loops Phase 2's per-layer block)
• programming_examples/llms/verify/verify_runner.py — both lenses: _run_diagnosis (per-layer cosine) and the topk_token gate (compute_topk_set_check)
• programming_examples/llms/verify/README.md — the verify methodology (HF bf16 reference, top-k token-set gate, cosine as diagnosis)
• <model>/docs/development_progress/ + the kernel_registry "tested shapes" rows with Used by = <model> — Phase 1 verified kernels (Phase 3 confirms they compose at scale)

Workflow

Step 1: Wire all N layers

Implement run_full_prefill(input_ids, weights, config) (or reuse the deployment's run_npu_prefill):

• Kernel-first path (default): loop run_transformer_block_<model>(...) (the per-model block runner Phase 2 produced) for layer_idx in range(config.n_layers), then apply final RMSNorm + LM head.
• Inheritance path (shortcut, bit-for-bit match only): loop llama32_1b_prefill.run_transformer_block(...) the same way.

Either way, this is just iterating the Phase 2 block runner N times plus head — no new kernels.

Step 2: Canonical prompts

Use the deployment's verify prompt set (verify/prompts/{base,instruct}.txt — base for base checkpoints, instruct for instruct/chat checkpoints). make verify runs 2 prompts × 32 tokens by default; make verify-full runs the full set. make diagnosis runs a single prompt for the per-layer cosine table.

The token-set gate handles the old "decisive vs competitive prompt" distinction automatically: a top-1 flip within the top-5 band is treated as benign drift (not a failure), while a token leaving the top-5 entirely is a failure. No manual per-prompt probability classification is needed.

Step 3: Run both lenses + collect metrics

cd programming_examples/llms/<model>
flock -x -w 1800 /tmp/mlir-air-npu.lock make diagnosis    # per-layer cosine, all layers
flock -x -w 1800 /tmp/mlir-air-npu.lock make verify       # token-set gate (exit 0/1)

From verify: PASS/FAIL (criterion 1, the gate); the report under verify/reports/ records the first divergence and the top-5 sets on each side. From diagnosis: read the per-layer cosine table as the localization lens (criterion 2) — not a pass/fail, but where you look when verify fails.

Step 4: Bisect on FAIL

If make verify fails, the diagnosis cosine table localizes where:

• Sudden cliff at layer i (cos[i] >> cos[i+1]) → layer-indexed bug at i+1 (weight load shifted, wrong bo_key, wrong wq for that layer)
• Gradual drift but a layer < 0.85 → BF16 accumulator saturating; check whether the production GEMM path uses F32 internal accumulate; cross-check make verify — if the token-set gate still PASSES, the drift is geometric not a bug; if it FAILS, real bug
• Layer 0 already low → integration error in the single-block runner itself; revisit Phase 2

If make verify fails but per-layer cosine looks fine, the divergence is likely in the decode path / KV-cache (verify generates 32 tokens; the per-layer cosine only probes prefill). Validate KV cache values at end of prefill against HF. Within an offending layer, bisect kernel-by-kernel using Phase 2's CPU-fallback technique.

Failure modes

Symptom	Likely cause	Where to look
Per-layer cos cliff at one layer	Layer-indexed weight load bug or wrong bo_key=f"kernel_L{i}"	Print weight shapes per layer; compare K/V cache layout at boundary
Per-layer cos drifts gradually < 0.85 by last layer	Real BF16 saturation, OR Down GEMM missing F32 accumulator	Run make verify — if token-set gate PASSES, drift is geometric not a bug; otherwise check F32 accumulator pattern
Per-layer cos OK but make verify FAILS at prediction	LM head precision drop (BF16 truncation in vocab projection)	Apply F32 accumulator pattern to LM head GEMM
make verify FAILS: NPU top-1 NOT in HF top-5 at all	Real correctness bug	Step 4 bisect
NaN in output	Uninitialized BO / reused stale buffer	Invoke debug-bo-corruption
Per-layer cos OK, prefill prediction OK, but multi-token generation diverges quickly	KV cache update bug at decode time (per-layer cosine only probes prefill; verify's 32-token generation catches it)	Validate KV cache values at end of prefill match HF

For any failure not in the table, invoke superpowers:systematic-debugging.

Update protocol

On Phase 3 PASS, this is the end-to-end correctness milestone — NPU is now numerically faithful to the HF bf16 reference at full N-layer scale. Update:

• <model>/docs/development_progress/phase3_full.md: per-layer cos table (diagnosis) + make verify PASS per prompt
• <model>/docs/development_progress/progress.md: Phase 3 summary
• <model>/TODO.md: mark Phase 3, advance to perf phases

Phase 4 (prefill perf) and Phase 5 (decode perf) MUST preserve these gates (make diagnosis per-layer cosine + make verify token-set) after every optimization — perf cannot trade correctness.