phase-7-independent-evaluator

name

phase-7-independent-evaluator

description

Phase 7 of LLM deployment — spawn a fresh subagent that treats the deployment as UNTRUSTED, audits the `make verify` implementation (anti-reward-hacking: confirms the token-set gate runs the production path vs HF bf16), then re-runs it as the primary gate. Produces a structured `evaluation_report.md` a human can read in 2 minutes to know the full deployment state. Invoke as `/phase-7-independent-evaluator <model_dir>` or auto-spawn from deploy-new-llm after Phase 6 PASS.

Purpose

The deploy-new-llm chain is autonomous. Phase 6 wires up a make verify gate that's supposed to compare NPU vs the HF bf16 reference via the verify/ subsystem's top-k token-set check. But the deployment agent wrote (or copied) both the gate AND the production code it exercises — nothing structurally prevents the gate from being mocked, pointed at a stale baseline, or wired to a verify-only code path that bypasses the real kernels. Phase 7 closes this by:

1. Spawning a FRESH subagent with no context from the deployment
2. Auditing the make verify code path (anti-reward-hacking) — confirming the gate actually drives the production kernels vs HF bf16 and gates on the token-set check, not a mocked / stale / verify-only path. This is what makes the independent re-run meaningful: without it, "the agent re-ran the gate it wrote itself" proves nothing.
3. Re-running the end-to-end make verify independently — the shared verify/ top-k token-set check (vLLM-aligned) of NPU vs HF bf16 on the real production prefill/decode path. A clean independent PASS, on a gate the audit confirmed is real, is the verdict.
4. Producing a short evaluation report humans can read in 2 minutes.

The deployment is NOT considered trustworthy until this report is written and its overall verdict is PASS or PASS-with-warnings.

Phase 7 PASS criteria (HARD GATES)

1. make verify audited (anti-reward-hacking): subagent has read the Makefile target + the model's verify_adapter.py + the shared programming_examples/llms/verify/{verify_runner,comparators,runners/hf_runner}.py, and confirmed the gate (a) drives the model's PRODUCTION run_npu_prefill / run_npu_decode_step via the NpuRunner (not a mock), (b) compares against HF transformers in bf16 (torch_dtype=torch.bfloat16), (c) gates on compute_topk_set_check (top-5 set inclusion at first divergence), not a hardcoded return True / > 0 threshold.
2. make verify PASSES under the fresh subagent run: token-set gate exits 0 (no npu_vs_hf FAIL), on the audited-real gate.
3. Short evaluation report written with the audit verdict + the re-run result + the measured numbers behind them. Verdict PASS or PASS-with-warnings.

If the subagent declares PASS without showing measured numbers OR without auditing the verify code path, the report is REJECTED — re-spawn with stricter instructions.

Knowledge base references

• programming_examples/llms/verify/README.md — the verify methodology the audit checks against (HF bf16 reference, top-k token-set gate, cosine-as-diagnosis)
• programming_examples/llms/verify/comparators.py — the gate implementation (compute_topk_set_check) the audit confirms is real
• <model>/docs/development_progress/ + the kernel_registry rows with Used by = <model> — Phase 1 catalog (subagent compares its measurements against this)
• programming_examples/kernel_registry/supported_kernels.md — kernel-by-kernel ground truth

Workflow

Step 1: Spawn fresh subagent with independence + audit constraints

Use the general-purpose Agent type. Critical instructions in the spawn prompt:

• Independence: do NOT read <model>/docs/development_progress/{LESSONS,progress,phaseN_*}.md BEFORE forming your own measurements. You may CITE them AFTER measuring (compare your numbers vs claimed).
• Re-derivation: every PASS/FAIL verdict must be backed by a number YOU measured or a code path YOU read during this audit, not a number copied from a deployment doc.
• Reward-hacking smell test: if the deployment-claimed numbers look surprisingly good, audit the gate implementation FIRST — does make verify really drive the production prefill/decode vs HF bf16 and gate on the token-set check, or did the agent shortcut to "if NPU output exists → PASS" / point the verify_adapter NpuRunner at a mock / compare against a stale cached baseline?

Step 2: Audit make verify — anti-reward-hacking

BEFORE running anything, READ:

1. The Makefile: what command does make verify invoke?

   grep -A 5 "^verify:" <model_dir>/Makefile
   

   It should run the shared runner ../verify/verify_runner.py --runner=<model>.verify_adapter --prompts topk_token ... — not a model-local copy of the runner.
2. <model_dir>/verify_adapter.py: this is the model's only verify code. Confirm its NpuRunner (the .prefill() / .decode_step() methods) imports and calls THIS model's production run_npu_prefill / run_npu_decode_step from <model>_inference.py — NOT a stub, NOT the llama32_1b copy. (The runner/comparators/report/HF runner live once in the shared programming_examples/llms/verify/; the adapter is the per-model hook.)
3. programming_examples/llms/verify/runners/hf_runner.py (shared): confirm the reference loads with AutoModelForCausalLM.from_pretrained(..., torch_dtype=torch.bfloat16).
4. programming_examples/llms/verify/comparators.py + programming_examples/llms/verify/report.py (shared): confirm the gate is compute_topk_set_check (top-5 set inclusion at first divergence) and report.has_failure() drives a real exit code — not a hardcoded return True / > 0 threshold.
5. If any of (2)-(4) fails: report [FAIL] verify gate is reward-hacked and stop here. Tag the deployment as needs-remediation.

Step 3: Run make verify — the PRIMARY gate

cd <model_dir>
flock -x -w 1800 /tmp/mlir-air-npu.lock make verify

Expected (per Phase 6 / verify subsystem design):

• Both NPU and HF bf16 greedy-decode the prompt set × 32 tokens.
• At the first divergence, NPU's chosen token ∈ HF top-5 AND HF's chosen token ∈ NPU top-5 → PASS; exit 0; no npu_vs_hf FAIL in the report.
• The report written under the model's build dir (reports/, by the shared programming_examples/llms/verify/ runner) records the first divergence + top-5 sets.

Also run make diagnosis to capture the per-layer cosine table as an informational sanity signal — eyeball it for a gross cliff or a NaN layer. It is NOT a gate: the verify subsystem retired threshold-based diagnosis (compare_pair reports cosine with no pass/fail), so do not fail the evaluation on a cosine number. make verify (the token-set gate) is the sole binding numeric verdict; the cosine table just helps localize if verify fails.

If the gate fails: record exact failure + cite the divergence position and which token left the top-5. Verdict = FAIL.

Step 4: Write the evaluation report

Output: <model_dir>/docs/evaluation_report.md. Keep it short — its job is to let a human know, in 2 minutes, whether the deployment is trustworthy and why. Use the structure below:

# Evaluation Report: <Model> on NPU2

## Verdict: <PASS / PASS-with-warnings / FAIL>  (<date>)

## 1. Gate audit (anti-reward-hacking)
- `make verify` drives production `run_npu_prefill` / `run_npu_decode_step`
  via `<model>/verify_adapter.py` NpuRunner (not a mock): <yes/no + evidence>
- Reference is HF transformers bf16 (`torch_dtype=torch.bfloat16`): <yes/no>
- Gate is `compute_topk_set_check` (top-5 at first divergence), not a
  hardcoded pass / `> 0` threshold: <yes/no>
- Conclusion: gate is REAL / reward-hacked

## 2. Independent `make verify` re-run
- Result: PASS / FAIL — first divergence at token <i>; NPU & HF chosen
  tokens both in the other's top-5
- `make diagnosis` per-layer cosine (informational only): <X→Y; cliff/NaN?>

## 3. Manual reproduce
    cd <model_dir>
    flock -x -w 1800 /tmp/mlir-air-npu.lock make verify

Failure modes

Symptom	Likely cause	What to do
Step 2 audit fails — make verify doesn't drive the production path vs HF bf16	Reward-hacked gate (mock NpuRunner, stale baseline, hardcoded pass)	Report [FAIL] verify gate is reward-hacked; deployment needs remediation; do NOT mark PASS
make verify returns PASS suspiciously fast (<10 s for a full N-layer model)	Token count too low, or NpuRunner not actually invoking kernels	Check the shared verify_runner.py GATE_N_TOKENS + the model's NpuRunner generates 32 tokens through the real prefill/decode
Subagent reads LESSONS/progress before measuring	Skill prompt wasn't strict enough	Reject the report; re-spawn with stricter instructions

For any failure not in the table, invoke superpowers:systematic-debugging.

Update protocol

This is the terminal verification phase. On Phase 7 PASS or PASS-with-warnings:

• <model_dir>/docs/evaluation_report.md is the durable artifact
• Append to <model>/TODO.md: "Independently evaluated YYYY-MM-DD: "
• Reference the report from <model>/docs/development_progress/progress.md

If FAIL: deployment cannot be tagged. Issues must be remediated and Phase 7 re-run.