메뉴
Verify whether a coding agent's patch correctly fixes a bug. Scores patches on structured criteria (problem alignment, minimality, logic correctness, scope, completeness). Use when selecting the best patch from N candidates or filtering low-quality fixes.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Plan, execute, and analyze LLM serving benchmarks across vLLM and SGLang configurations. Use when user says "run benchmarks", "benchmark config", "QPS sweep", "compare serving configs", "custbench", "benchmark analysis", or "generate benchmark report". Do NOT use for writing Terraform (use terraform-automation), deployment validation (use deployment-orchestrator), or GPU hardware diagnostics (use gpu-infra-troubleshooting).
Launch and manage detached agent runs on EKS via the agent-runner CLI. Use when the user says "launch an agent run", "agent runtime", "agent-runner", "run this spec on the cluster", "check run status", "agent run logs", or "stop a run". Do NOT use for writing the agent-runner code itself (edit the sibling ../agent-runner repo), Terraform of the foundation (use terraform-automation), or GPU serving deployments (use infra-deployer).
Evaluate upstream PRs or commits individually against the K2.6 baseline. Tests each change in isolation using the same L0-L4 pipeline, identifying which are helpful, harmful, or neutral for our specific workload. Use when new upstream PRs land (Alpha-MoE, Mega MoE, FP8 fusion) or when evaluating a bundle of commits. Never assume a bundle is net-positive — the EFA harness found bundled expert commits regressed LL performance by 10%.
Analyze a failed or underperforming kernel candidate using ncu profile data and telemetry history. Produces a structured diagnosis (what's wrong, why, what to try next) and updates the constraint database. Use after /verify-kernel returns a failure or an L3/L4 regression, or when the agent needs guidance on which optimization direction to pursue.
Generate an optimized kernel candidate using the current state vector approach. Injects all active constraints from the database, uses the specified DSL (Triton/TileLang/CUDA), and targets the highest-headroom region identified by profiling. Use after profiling identifies a target, or when the previous candidate fails and the state vector has advanced.
View, query, add, or demote constraints in the kernel optimization constraint database. The database is append-only — constraints are never deleted, only demoted in severity if contradicted by evidence. Use to inspect what the agent has learned, manually inject domain knowledge, or query relevant constraints before generating a candidate.
| name | patch-verifier |
| description | Verify whether a coding agent's patch correctly fixes a bug. Scores patches on structured criteria (problem alignment, minimality, logic correctness, scope, completeness). Use when selecting the best patch from N candidates or filtering low-quality fixes. |
Scores agent-generated patches against a structured rubric to predict whether the patch will pass gold tests. Designed for post-generation verification — the agent has already produced a diff, and this skill evaluates it.
Given a problem statement and a patch diff, load the active rubric version from versions/ and score the patch.
git diff output from the agentversions/v*.md to use (default: latest)Structured JSON with per-criterion scores (0.0-1.0), overall score, confidence, verdict, and reasoning.
scripts/verify_patch.py — Core verification: calls Claude API with rubric + patch → structured scorescripts/sweep_versions.py — Batch evaluation: all versions × all patches → telemetry JSONLscripts/analyze_versions.py — Compare versions on precision/recall/ECE/AUROCscripts/analyze_errors.py — Post-sweep error analysis: FP/FN categorization, failure mode matching, recommendationsscripts/smoke_test.py — Pre-sweep validation: 4 known-outcome patches, directional correctness checksscripts/telemetry.py — Telemetry event emitter and circuit breaker logicscripts/preprocess_diff.py — Diff preprocessor: strips cosmetic hunks for oversized diffsscripts/run_iteration.py — Utility: smoke test → sweep → error analysis → holdout evalscripts/verify_patch_ensemble.py — Production ensemble: v001∩v009(2+ lc), precision=1.00scripts/describe_then_verify.py — Two-stage for FM-001: describe expected fix → search large diff (experimental)| Version | Description | Addresses |
|---|---|---|
| v001_baseline | 6-criteria rubric | — |
| v002_minimal | 3-criteria ablation | — |
| v003_fewshot | Baseline + 2 labeled examples | — |
| v004_cot | Chain-of-thought scoring | — |
| v005_binary | Binary yes/no, no rubric | — |
| v006_reformat_aware | Reformatting-aware rubric | FM-001 |
| v007_strict_logic | Strict logic tracing, conservative | FM-003 |
| v008_conservative_reformat | Conservative + reformatting-aware | FM-001 + FM-003 |
| v009_adversarial | Adversarial bug-finding rubric | FM-003 (breakthrough) |
| ID | Name | Description |
|---|---|---|
| FM-001 | Reformatting noise | Large diffs dominated by style changes hiding small functional fixes. Addressed by v006. |
| FM-002 | Truncated diffs | Diffs missing trailing newline. Fixed in gold_eval.py. |
| FM-003 | Plausible but wrong | Patch targets right area, looks correct, but has subtle logic error. Verifier gives high scores (0.85-0.98). 79% of Phase 2b errors. |
| FM-004 | Diff truncation | 47% of diffs exceeded 12K char limit. 3/6 passing patches invisible to verifier. Fixed: 100K limit + preprocess_diff.py. |
references/rubric_design.md — Design rationale and iteration historyversions/ — Immutable rubric versions (never edit, only create new)