Pre-adoption evaluation recipe for any third-party ML artifact (router, classifier, embedding model, RAG pipeline). Before recommending adoption — or even running it on real production traffic — load this skill and run the 6-gate checklist: (1) README-vs-claim triangulation, (2) code-level load-path inspection, (3) shipped-checkpoint-vs-claimed-class-count gate (the silent killer — 2026-07-22 incident: NUM_LABELS=4 in code, NUM_LABELS=6 in shipped weights), (4) per-turn architecture boundary check (truncation, fallback policy, confidence threshold), (5) on-distribution label-coverage audit against YOUR real workload, (6) realistic cost-cut estimate for YOUR workload (not the vendor's headline figure).
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Pre-adoption evaluation recipe for any third-party ML artifact (router, classifier, embedding model, RAG pipeline). Before recommending adoption — or even running it on real production traffic — load this skill and run the 6-gate checklist: (1) README-vs-claim triangulation, (2) code-level load-path inspection, (3) shipped-checkpoint-vs-claimed-class-count gate (the silent killer — 2026-07-22 incident: NUM_LABELS=4 in code, NUM_LABELS=6 in shipped weights), (4) per-turn architecture boundary check (truncation, fallback policy, confidence threshold), (5) on-distribution label-coverage audit against YOUR real workload, (6) realistic cost-cut estimate for YOUR workload (not the vendor's headline figure).
["evaluate this model","evaluate this router","evaluate this classifier","is this good for our use case","should we adopt this","replay our convos through this","does it pass for classification","check if the claims hold","vendor model eval","third-party router eval","run it on my past convos","what would it choose on my data","are the choices right","eval before adoption","pre-adoption check"]
changelog
["0.1.0 (2026-07-22): Initial version. Recipe distilled from `deycoding/deycoding-compliance-classifier-router` evaluation — the first end-to-end run of this checklist. Six gates + the headline finding pattern (NUM_LABELS=4 vs shipped-checkpoint head=6) that the eval surfaced. Companion to `finish-the-job` (the eval-stall anti-pattern was patched there). Verified end-state: 11 prompts run against the actual model, 441MB weights loaded from HuggingFace, eval report at `~/projects/router-eval/EVAL_REPORT.md`."]
Pre-adoption evaluation recipe for third-party ML artifacts. Run this skill when a user asks to evaluate / adopt / replay-against-real-data / check-claims-hold on a third-party model, router, classifier, embedding, or pipeline. The class-of-failure mode this skill catches: vendor claims that pass the headline test but break under code-level inspection.
Why this skill exists
Headline claims ("99.2% accuracy", "65-73% cost savings", "<10ms latency", "supports whole-convo classification") are the first layer of evaluation, not the last. The 2026-07-22 incident (this skill's originating event) had every headline claim — and the shipped repo does not run end-to-end as-documented. Specifically:
server/server.py sets NUM_LABELS = 4 and a 4-entry LABELS dict (simple_no_pii, simple_pii, complex_no_pii, complex_pii).
The shipped state_dict from HuggingFace has a 6-class classification head (classifier.3.weight: torch.Size([6, 768])).
The README documents the architecture as Linear(768-768) - Tanh - Dropout - Linear(768-4).
Three sources of truth disagree. The FastAPI server cannot start; load_state_dict raises on shape mismatch with the default strict=True.
This skill catches that class of failure systematically, before recommending adoption. The same checklist catches: short-context truncation that makes "whole-convo" impossible, fallback policies that mask real ambiguity, label sets that don't cover the user's actual distribution, and headline cost-savings numbers that only hold for the vendor's narrow workload.
When to load
Load this skill when any of these signals fire:
Signal
Example
User shares a third-party ML artifact and asks "is this good"
User asks to replay their data through a third-party classifier
"can we replay it against some of my older convos"
User asks for adoption guidance on a vendor router
"should we adopt this for our cost-cut"
User asks to verify vendor claims
"do the claims hold / does it actually do what they say"
User shares a Medium / blog post about a vendor's ML pipeline
"look at this article and tell me if it would work for us"
Do not load this skill for: evaluating your own model (use verify-deployed-frontend-fix or the harness-engineering skill), CLI tool env var claims (use cli-env-var-verification), sibling-bug triage on your own campaign (use convergent-bug-triage).
The 6-gate checklist
Run these in order. Each gate is independently load-bearing — a fail at gate N invalidates downstream gate results.
Gate 1 — README-vs-claim triangulation
Goal: Verify the artifact's docs match its claims. Even before reading code, mine the README (or article, or paper) for:
Architecture numbers (parameters, dimensions, layers, max sequence length, vocabulary size)
Cost-savings claims (typically stated as a percentage on the vendor's workload)
Label set / class taxonomy (what the model is supposed to output)
Deployment story (one command? CloudFormation? Docker? local-only?)
Write these into a 1-row table at the top of the eval report. You'll cross-reference each claim against the code at Gate 3.
Pitfall: Medium / blog articles often lose nuance that the README preserves. If the article is bot-walled or paywalled, the README IS the article — read it as the canonical source. Do not ask the user to paste the article.
Gate 2 — Code-level load-path inspection
Read the actual file(s) that load and use the model. For PyTorch: model = SomeClass(...) + model.load_state_dict(torch.load(MODEL_PATH, ...)) + model.eval(). For HuggingFace transformers: AutoModel.from_pretrained(...) + AutoTokenizer.from_pretrained(...). For ONNX: ort.InferenceSession(...). For TFLite: tf.lite.Interpreter(...).
Extract and tabulate:
Field
Value
Source
NUM_LABELS / num_classes
integer
constants or config
MAX_SEQ_LEN / max_position_embeddings
integer
config dataclass or argparse
LABELS dict / label mapping
name → index
constants or yaml
MODEL_PATH env var default
string path
env var block
TOKENIZER_PATH env var default
string path
env var block
CONFIDENCE_THRESHOLD default
float
agent config
FALLBACK_LABEL default
string
agent config
device resolution
cuda / cpu / mps
runtime check
Gate 3 — Shipped-checkpoint-vs-claimed-class-count gate (the silent killer)
This gate is the load-bearing one. It catches the NUM_LABELS=4 in code, NUM_LABELS=6 in weights class of bug. The recipe:
import torch
sd = torch.load('<model_path>', map_location='cpu', weights_only=False)
# Find the classifier head's weight tensor (last linear layer's W)# Typical names: 'classifier.3.weight', 'classifier.classifier.weight',# 'head.weight', 'output.weight', 'pooler.dense.weight'for k, v in sd.items():
if'classifier'in k.lower() or'head'in k.lower() or'output'in k.lower():
if v.ndim == 2:
print(k, v.shape) # should be (NUM_CLASSES, HIDDEN_DIM)break
If the head shape does NOT match the claimed NUM_LABELS, the repo is broken end-to-end. The FastAPI server will raise on load. Surface this in the verdict as a 🔴 red finding, not as a paper-over. Two possible resolutions:
Update NUM_LABELS in code to match the checkpoint. This unblocks inference but breaks the documented label mapping — you don't know what indices 4, 5, etc. mean unless the vendor published a separate label dictionary.
Re-train or re-export the checkpoint to match the documented label set. This unblocks the documented label mapping but requires the vendor's training code.
Either way, do not adopt the repo as-is. Surface this to the user before any deployment.
Gate 4 — Per-turn architecture boundary check
Most vendor routers are per-turn, not per-conversation. Verify by checking:
MAX_SEQ_LEN / max_position_embeddings / truncation defaults — if it's small (e.g. 128, 512), the model silently truncates anything longer.
The API contract (POST /classify or POST /chat or POST /route) — does it accept a single query, a list of queries, or a multi-message conversation?
The model architecture — bidirectional encoders (BERT, RoBERTa, MiniMax-M3-embed) are per-segment. Causal decoders (GPT, LLaMA) are autoregressive. Long-context variants (Longformer, BigBird, RoPE-extended) are the only ones that can do whole-convo.
If the user asked "does it pass in the whole convo" and the architecture is per-turn, the answer is no, by design. That's not a bug; that's the wrong product. Don't recommend a per-turn router for a whole-convo problem; recommend a different product.
Gate 5 — On-distribution label-coverage audit
The vendor's published label set may not match the user's actual task distribution. Mine the user's past session_search for ~30-100 recent real prompts and bucket them by task type. Compare that distribution against the vendor's class taxonomy.
Example from the 2026-07-22 incident: the vendor's 4-class taxonomy was {simple_no_pii, simple_pii, complex_no_pii, complex_pii}. The user's real distribution was roughly {git/CLI ops, Slack digests, AO orchestration, CSS/UI, PR review, beads, memory/skill management}. Zero overlap. Even if the model classifies perfectly, the routing decisions don't help — a gh pr view query routed to "small model" is wrong because the user already prefers the long-context model for that query type.
Recipe:
Mine session_search for the user's top 5-10 active projects over the last 30 days.
Extract the first user prompt from each session.
Bucket them by hand-labeled task type (you can use a quick LLM call to label them in batches, or eyeball it for 30-50 prompts).
Compare the bucket distribution to the vendor's label taxonomy.
Flag as 🟡 if <30% of user tasks have a natural mapping to a vendor class; flag as 🔴 if <10%.
Gate 6 — Realistic cost-cut estimate for YOUR workload
Vendor headline numbers are measured on the vendor's workload, not yours. The 60% cost cut on deycoding-compliance-classifier-router is a BFSI banking-chat figure — your dev-ops / Slack-digest / browser-automation distribution won't see the same savings.
Recipe for a realistic estimate:
Look at the vendor's cost-cut derivation: it's typically (small-model-cost * %-small-traffic) + (large-model-cost * %-large-traffic) - routing-overhead.
Estimate your %-small-traffic by applying Gate 5's bucket distribution to the vendor's small/large routing rules.
Estimate routing overhead (typically 5-15ms per request).
Compare against your current cost distribution (large model for everything → 100% large cost).
Realistic savings on a non-matching workload: 5-20%, not the vendor's headline.
Be honest with the user about this. A 60% saving on the vendor's workload is a marketing number; a 10-20% saving on yours is the engineering reality. Do not adopt based on the vendor's figure.
Verdict shape (mandatory)
Every eval run ends with a verdict in this shape:
🔴 / 🟡 / 🟢 verdict on whether to adopt
- Headline finding: <the single most damning or interesting thing the eval turned up>
- Gate-by-gate: 1-6 with pass/fail and one-line evidence
- Realistic cost-cut for your workload: <number>%
- Recommendation: adopt / fork-and-patch / do-not-adopt / build-our-own
- Proof: <path to eval report, raw logits, log dump>
If the verdict is "do not adopt" or "fork-and-patch", also include a one-line "what would change my mind" or "what fork-and-patch requires". The user wants to know the minimum delta that would make adoption viable.
Anti-patterns (do not do)
❌ "Trust the README." The README is the first source of truth, not the only one. Always triangulate with code + weights + behavior.
❌ "Trust the headline accuracy." Headline accuracy is measured on the vendor's test set. Your distribution is different. Gate 5 catches this.
❌ "Skip Gate 3 because the README says it works." Gate 3 is the silent killer. The 2026-07-22 incident proves a vendor can ship a README that documents the wrong number of classes while the shipped weights have the right (different) number.
❌ "Classify the user's data and report accuracy." Without Gate 5's label-coverage audit, the accuracy number is meaningless — you're measuring how well the model classifies into a taxonomy the user doesn't actually use.
❌ "Recommend adoption if the architecture sounds right." Architecture rightness ≠ workload rightness. The same router can be production-grade for BFSI and a bad fit for engineering dev work.
❌ "Stall on a pre-execution menu." When the user says "evaluate this, replay my convos, are the choices right", execute the verbs in sequence. See finish-the-jobreferences/eval-stall-on-preflight-menu-2026-07-22.md.
Support files
scripts/check_checkpoint_class_count.py — Gate 3 reproducer. Detects PyTorch / HuggingFace classifier heads, walks the state_dict or config, and reports (or asserts, with --expected N) the class count. Catches the 2026-07-22 incident's class of bug in one command: python3 scripts/check_checkpoint_class_count.py <model_path> --expected 4. Exit 0 = pass, exit 1 = mismatch.
Cross-references
finish-the-job — the eval-stall anti-pattern (pre-execution menu) was patched there with a companion reference.
cli-env-var-verification — binary-string grep on compiled executable + official vendor docs + live process test + config-file presence. Same triangulation discipline, applied to env vars instead of model artifacts.
convergent-bug-triage — for when the eval surfaces multiple sibling issues (e.g. three separate failures all traceable to the same root cause like the class-count mismatch).
pr-description-validator-gate6b — same gate discipline (read-the-validator-locally, don't trust the headline PASS/FAIL summary), applied to PR description validators instead of model artifacts.
Worked example — the 2026-07-22 incident
User asked: "clone this locally and evaluate it. Does it pass in the whole convo for classification? Can we replay it against some of my older convos/projects to see what it would choose and if the choices right?" with a Medium link.
Phase 0: Classified as vendor-ml-artifact-eval (no existing skill covered this class). Loaded this skill.
Gate 2 (code-level load path): server/server.py:18-19 shows NUM_LABELS = 4; LABELS = {0: "simple_no_pii", 1: "simple_pii", 2: "complex_no_pii", 3: "complex_pii"}. MAX_SEQ_LEN = 128 at line 17. model.load_state_dict(torch.load(MODEL_PATH, ...), weights_only=False) at line 118 — note weights_only=False (security smell, but not the issue here).
Gate 3 (checkpoint-vs-claimed-class-count): Loaded the model weights from HuggingFace. The final classification layer classifier.3.weight has shape torch.Size([6, 768]). MISMATCH. The shipped code wants NUM_LABELS=4 but the shipped weights have a 6-class head. load_state_dict will raise. 🔴 Block.
Gate 4 (per-turn architecture boundary): MAX_SEQ_LEN=128 tokens. README example for Design fraud detection system for UPI transactions (8 tokens) fits; Read my convo history to see what wasn't optimal... (65 tokens) fits but is heavily truncated; a full conversation thread (2000-4000 words) does NOT fit. The model cannot do whole-convo classification. Different product.
Gate 5 (on-distribution label coverage): Mined 8 real past Jeffrey prompts via session_search. Buckets: git/CLI ops (3), AO orchestration (2), Slack digests (1), PR work (1), other (1). Vendor's 4-class taxonomy (complexity × PII) has zero direct overlap with this distribution. Routing a git push origin <branch> to "small model" because it has no PII is wrong.
Gate 6 (realistic cost-cut estimate): Vendor's 60% figure is BFSI-derived. Engineering dev workload (long-context code review, file diff analysis, multi-file refactors) overwhelmingly wants the long-context model — the small-model tier wouldn't even fit most past prompts in its 128-token context window. Realistic estimate for the user's actual distribution: 10-20%, not 60%.
Verdict: 🟡 workable compliance-routing primitive; does not, and cannot, do whole-conversation classification as the user asked; shipped code and shipped weights are out of sync; not ready for drop-in. Recommend fork + re-train on a task-type taxonomy matching the user's session distribution (~2-week AO dispatch).
Proof: Eval report at ~/projects/router-eval/EVAL_REPORT.md (16 KB). Raw logits at /tmp/router_eval_results.json. Model weights + tokenizer at ~/projects/router-eval/deycoding-compliance-classifier-router/server/deycoding.compliance-classifier-in-1-0.{pt,json}. Reproducer recipe documented in the report.
