# Phase 7 Independent Evaluator > Purpose - Skill: `xilinx/phase-7-independent-evaluator` (Agent Skill) - Install (CLI): `npx skillmds@latest add xilinx/phase-7-independent-evaluator` - Raw SKILL.md: https://api.skillmd.com/api/skills/xilinx/phase-7-independent-evaluator/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: xilinx (https://skillmd.com/u/xilinx) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/xilinx/phase-7-independent-evaluator --- ## Purpose The deploy-new-llm chain is autonomous. Phase 6 wires up a `make verify` gate that's supposed to compare NPU vs the HF bf16 reference via the `verify/` subsystem's top-k token-set check. But the deployment agent wrote (or copied) both the gate AND the production code it exercises — nothing structurally prevents the gate from being mocked, pointed at a stale baseline, or wired to a verify-only code path that bypasses the real kernels. Phase 7 closes this by: 1. **Spawning a FRESH subagent** with no context from the deployment 2. **Auditing the `make verify` code path** (anti-reward-hacking) — confirming the gate actually drives the production kernels vs HF bf16 and gates on the token-set check, not a mocked / stale / verify-only path. This is what makes the independent re-run meaningful: without it, "the agent re-ran the gate it wrote itself" proves nothing. 3. **Re-running the end-to-end `make verify` independently** — the shared `verify/` top-k token-set check (vLLM-aligned) of NPU vs HF bf16 on the real production prefill/decode path. A clean independent PASS, on a gate the audit confirmed is real, is the verdict. 4. **Producing a short evaluation report** humans can read in 2 minutes. The deployment is NOT considered trustworthy until this report is written and its overall verdict is PASS or PASS-with-warnings. ## Phase 7 PASS criteria (HARD GATES) 1. **`make verify` audited** (anti-reward-hacking): subagent has read the Makefile target + the model's `verify_adapter.py` + the shared `programming_examples/llms/verify/{verify_runner,comparators,runners/hf_runner}.py`, and confirmed the gate (a) drives the model's PRODUCTION `run_npu_prefill` / `run_npu_decode_step` via the `NpuRunner` (not a mock), (b) compares against HF transformers in bf16 (`torch_dtype=torch.bfloat16`), (c) gates on `compute_topk_set_check` (top-5 set inclusion at first divergence), not a hardcoded `return True` / `> 0` threshold. 2. **`make verify` PASSES under the fresh subagent run**: token-set gate exits 0 (no `npu_vs_hf` FAIL), on the audited-real gate. 3. **Short evaluation report written** with the audit verdict + the re-run result + the measured numbers behind them. Verdict PASS or PASS-with-warnings. If the subagent declares PASS without showing measured numbers OR without auditing the verify code path, the report is REJECTED — re-spawn with stricter instructions. ## Knowledge base references - `programming_examples/llms/verify/README.md` — the verify methodology the audit checks against (HF bf16 reference, top-k token-set gate, cosine-as-diagnosis) - `programming_examples/llms/verify/comparators.py` — the gate implementation (`compute_topk_set_check`) the audit confirms is real - `/docs/development_progress/` + the `kernel_registry` rows with Used by = `` — Phase 1 catalog (subagent compares its measurements against this) - `programming_examples/kernel_registry/supported_kernels.md` — kernel-by-kernel ground truth ## Workflow ### Step 1: Spawn fresh subagent with independence + audit constraints Use the `general-purpose` Agent type. Critical instructions in the spawn prompt: - **Independence**: do NOT read `/docs/development_progress/{LESSONS,progress,phaseN_*}.md` BEFORE forming your own measurements. You may CITE them AFTER measuring (compare your numbers vs claimed). - **Re-derivation**: every PASS/FAIL verdict must be backed by a number YOU measured or a code path YOU read during this audit, not a number copied from a deployment doc. - **Reward-hacking smell test**: if the deployment-claimed numbers look surprisingly good, audit the gate implementation FIRST — does `make verify` really drive the production prefill/decode vs HF bf16 and gate on the token-set check, or did the agent shortcut to "if NPU output exists → PASS" / point the `verify_adapter` NpuRunner at a mock / compare against a stale cached baseline? ### Step 2: Audit `make verify` — anti-reward-hacking BEFORE running anything, READ: 1. **The Makefile**: what command does `make verify` invoke? ```bash grep -A 5 "^verify:" /Makefile ``` It should run the **shared** runner `../verify/verify_runner.py --runner=.verify_adapter --prompts topk_token ...` — not a model-local copy of the runner. 2. **`/verify_adapter.py`**: this is the model's only verify code. Confirm its `NpuRunner` (the `.prefill()` / `.decode_step()` methods) imports and calls THIS model's production `run_npu_prefill` / `run_npu_decode_step` from `_inference.py` — NOT a stub, NOT the llama32_1b copy. (The runner/comparators/report/HF runner live once in the shared `programming_examples/llms/verify/`; the adapter is the per-model hook.) 3. **`programming_examples/llms/verify/runners/hf_runner.py`** (shared): confirm the reference loads with `AutoModelForCausalLM.from_pretrained(..., torch_dtype=torch.bfloat16)`. 4. **`programming_examples/llms/verify/comparators.py`** + **`programming_examples/llms/verify/report.py`** (shared): confirm the gate is `compute_topk_set_check` (top-5 set inclusion at first divergence) and `report.has_failure()` drives a real exit code — not a hardcoded `return True` / `> 0` threshold. 5. **If any of (2)-(4) fails**: report `[FAIL] verify gate is reward-hacked` and stop here. Tag the deployment as needs-remediation. ### Step 3: Run `make verify` — the PRIMARY gate ```bash cd flock -x -w 1800 /tmp/mlir-air-npu.lock make verify ``` Expected (per Phase 6 / verify subsystem design): - Both NPU and HF bf16 greedy-decode the prompt set × 32 tokens. - At the first divergence, NPU's chosen token ∈ HF top-5 AND HF's chosen token ∈ NPU top-5 → PASS; exit 0; no `npu_vs_hf` FAIL in the report. - The report written under the model's build dir (`reports/`, by the shared `programming_examples/llms/verify/` runner) records the first divergence + top-5 sets. Also run `make diagnosis` to capture the per-layer cosine table as an **informational** sanity signal — eyeball it for a gross cliff or a NaN layer. It is NOT a gate: the verify subsystem retired threshold-based diagnosis (`compare_pair` reports cosine with no pass/fail), so do not fail the evaluation on a cosine number. `make verify` (the token-set gate) is the sole binding numeric verdict; the cosine table just helps localize *if* verify fails. If the gate fails: record exact failure + cite the divergence position and which token left the top-5. Verdict = FAIL. ### Step 4: Write the evaluation report Output: `/docs/evaluation_report.md`. Keep it short — its job is to let a human know, in 2 minutes, whether the deployment is trustworthy and why. Use the structure below: ```markdown # Evaluation Report: on NPU2 ## Verdict: () ## 1. Gate audit (anti-reward-hacking) - `make verify` drives production `run_npu_prefill` / `run_npu_decode_step` via `/verify_adapter.py` NpuRunner (not a mock): - Reference is HF transformers bf16 (`torch_dtype=torch.bfloat16`): - Gate is `compute_topk_set_check` (top-5 at first divergence), not a hardcoded pass / `> 0` threshold: - Conclusion: gate is REAL / reward-hacked ## 2. Independent `make verify` re-run - Result: PASS / FAIL — first divergence at token ; NPU & HF chosen tokens both in the other's top-5 - `make diagnosis` per-layer cosine (informational only): ## 3. Manual reproduce cd flock -x -w 1800 /tmp/mlir-air-npu.lock make verify ``` ## Failure modes | Symptom | Likely cause | What to do | |---|---|---| | Step 2 audit fails — `make verify` doesn't drive the production path vs HF bf16 | Reward-hacked gate (mock NpuRunner, stale baseline, hardcoded pass) | Report `[FAIL] verify gate is reward-hacked`; deployment needs remediation; do NOT mark PASS | | `make verify` returns PASS suspiciously fast (<10 s for a full N-layer model) | Token count too low, or NpuRunner not actually invoking kernels | Check the shared `verify_runner.py` GATE_N_TOKENS + the model's NpuRunner generates 32 tokens through the real prefill/decode | | Subagent reads LESSONS/progress before measuring | Skill prompt wasn't strict enough | Reject the report; re-spawn with stricter instructions | For any failure not in the table, invoke `superpowers:systematic-debugging`. ## Update protocol This is the terminal verification phase. On Phase 7 PASS or PASS-with-warnings: - `/docs/evaluation_report.md` is the durable artifact - Append to `/TODO.md`: "Independently evaluated YYYY-MM-DD: " - Reference the report from `/docs/development_progress/progress.md` If FAIL: deployment cannot be tagged. Issues must be remediated and Phase 7 re-run.