菜单
LLM-as-a-judge evaluations over OpenAI-compatible APIs (including NVIDIA Nemotron & LM Studio). Use whenever evaluating output quality, correctness, or scoring responses using standard rubrics.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | openai-judge |
| description | LLM-as-a-judge evaluations over OpenAI-compatible APIs (including NVIDIA Nemotron & LM Studio). Use whenever evaluating output quality, correctness, or scoring responses using standard rubrics. |
| validator_version | 2.0 |
| compatibility | python>=3.10 |
| version | 1.1.0 |
Perform LLM-as-a-judge scoring end to end: take prompt, rubric, and model endpoint, produce score and reasoning, and prove it worked before reporting success.
Confirm these hold before doing anything:
--mock is specified for local-only validation).NVIDIA_API_KEY for Nemotron).result.json to the output directory.Prefer using the execution script:
python scripts/run.py --model nvidia/nemotron-3-ultra-550b-a55b --prompt prompt.txt --rubric rubric.txt --out result.json
result.json is written.score and reasoning.You are not done until this exits 0:
python scripts/validate_skill.py --skill . --tier structural,behavioral
Example 1
Input: prompt.txt (What is 2+2?), rubric.txt (Grade answer correctness)
Output: result.json ({"score": 1.0, "reasoning": "The answer is correct."})
Detect and block architecture drift in CI by comparing a codebase's actual Python import graph against a declared C4 component model. Use this whenever the user wants to enforce architecture in CI/CD, fail builds on undocumented dependencies, keep C4/Mermaid diagrams in sync with code, set up architecture tests or fitness functions, prevent layering violations, or guard/lint/gate their architecture — even if they don't name C4 or Mermaid explicitly. Also use it to bootstrap an architecture manifest from an existing repo or to remediate a failing drift check.
Build, validate, and package reusable evaluation datasets for agent and LLM systems from prompts, traces, tool-call logs, metadata, and expected outcomes. Use this when the user asks to gather eval data, create benchmark datasets, normalize traces, define reusable ground truth, bootstrap eval suites, or produce retrieval, tool-invocation, response, or end-to-end evaluation artifacts.
Benchmark and A/B-test multiple LLMs or systems-under-test on one evaluation dataset. Use this whenever the user wants to compare models side by side, rank candidates by a score, run a multi-model comparison, set up a persistent A/B eval campaign, decide whether a new model is significantly better than a baseline, or produce a comparative HTML/JSON benchmark report.
Generates or updates C4 architecture documentation as Mermaid diagrams in the host repo (docs/architecture/ by default, or the repo's existing docs location). Produces Level 1 context and Level 2 container diagrams, plus Level 3 components only for the subsystem being changed. Updates existing diagrams in place and flags drift between diagrams and code instead of silently rewriting them.
Runs a security and quality review checklist over a diff or named files in an isolated forked subagent. Covers injection, secrets, authz/authn, error handling, resource leaks, concurrency, compatibility, test coverage, and hardcoded config. Returns severity-ranked findings with file:line and a blocking/non-blocking/clean verdict. Read-only - never edits code or proposes inline fixes.
Instantiates a constraint-programming planning template for a new task, producing a single markdown plan with five sections - objective function, feasible region, permission architecture, feedback loop, and context persistence. All success criteria must be mechanically verifiable (commands, not vibes).