Run any Skill in Manus with one click

openai-judge

Stars1

Forks0

UpdatedJuly 7, 2026 at 22:05

LLM-as-a-judge evaluations over OpenAI-compatible APIs (including NVIDIA Nemotron & LM Studio). Use whenever evaluating output quality, correctness, or scoring responses using standard rubrics.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

ianshank

ianshank/Agents

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

15 files

SKILL.md

readonly

More from this repository

same repository

architecture-drift-guard

ianshank/Agents

Detect and block architecture drift in CI by comparing a codebase's actual Python import graph against a declared C4 component model. Use this whenever the user wants to enforce architecture in CI/CD, fail builds on undocumented dependencies, keep C4/Mermaid diagrams in sync with code, set up architecture tests or fitness functions, prevent layering violations, or guard/lint/gate their architecture — even if they don't name C4 or Mermaid explicitly. Also use it to bootstrap an architecture manifest from an existing repo or to remediate a failing drift check.

2026-07-071

eval-corpus-forge

ianshank/Agents

Build, validate, and package reusable evaluation datasets for agent and LLM systems from prompts, traces, tool-call logs, metadata, and expected outcomes. Use this when the user asks to gather eval data, create benchmark datasets, normalize traces, define reusable ground truth, bootstrap eval suites, or produce retrieval, tool-invocation, response, or end-to-end evaluation artifacts.

2026-07-071

model-bench

ianshank/Agents

Benchmark and A/B-test multiple LLMs or systems-under-test on one evaluation dataset. Use this whenever the user wants to compare models side by side, rank candidates by a score, run a multi-model comparison, set up a persistent A/B eval campaign, decide whether a new model is significantly better than a baseline, or produce a comparative HTML/JSON benchmark report.

2026-07-071

c4-docs

ianshank/Agents

Generates or updates C4 architecture documentation as Mermaid diagrams in the host repo (docs/architecture/ by default, or the repo's existing docs location). Produces Level 1 context and Level 2 container diagrams, plus Level 3 components only for the subsystem being changed. Updates existing diagrams in place and flags drift between diagrams and code instead of silently rewriting them.

2026-07-031

code-review

ianshank/Agents

Runs a security and quality review checklist over a diff or named files in an isolated forked subagent. Covers injection, secrets, authz/authn, error handling, resource leaks, concurrency, compatibility, test coverage, and hardcoded config. Returns severity-ranked findings with file:line and a blocking/non-blocking/clean verdict. Read-only - never edits code or proposes inline fixes.

2026-07-031

plan

ianshank/Agents

Instantiates a constraint-programming planning template for a new task, producing a single markdown plan with five sections - objective function, feasible region, permission architecture, feedback loop, and context persistence. All success criteria must be mechanically verifiable (commands, not vibes).

2026-07-031

name	openai-judge
description	LLM-as-a-judge evaluations over OpenAI-compatible APIs (including NVIDIA Nemotron & LM Studio). Use whenever evaluating output quality, correctness, or scoring responses using standard rubrics.
validator_version	2.0
compatibility	python>=3.10
version	1.1.0

openai-judge — E2E Action Skill

Perform LLM-as-a-judge scoring end to end: take prompt, rubric, and model endpoint, produce score and reasoning, and prove it worked before reporting success.

1. Preconditions (input contract)

Confirm these hold before doing anything:

Target prompt text and grading rubric are present in files.
OpenAI-compatible model endpoint is reachable (or --mock is specified for local-only validation).
Environment variables or configurations are populated for authentication (e.g. NVIDIA_API_KEY for Nemotron).

2. Procedure (the E2E steps)

Read the input prompt and grading rubric from their target paths.
Initialize the OpenAI-compatible client wrapper with model endpoint and api key.
Call the completion API with the requested prompt and formatting instructions (preferring JSON output).
Extract the verdict JSON containing the score and reasoning.
Write result.json to the output directory.

Prefer using the execution script:

python scripts/run.py --model nvidia/nemotron-3-ultra-550b-a55b --prompt prompt.txt --rubric rubric.txt --out result.json

3. Output contract (postconditions — what "done" means)

Output file result.json is written.
Output is valid JSON with keys score and reasoning.
The score value is a float/integer representing the judgment.

4. Failure handling

On failure, clean up any partial files and raise a clear, descriptive error containing output/logs.
Gracefully handle JSON parsing errors by returning a structured default failure JSON.

5. Validation gate (before declaring success)

You are not done until this exits 0:

python scripts/validate_skill.py --skill . --tier structural,behavioral

6. Examples

Example 1 Input: prompt.txt (What is 2+2?), rubric.txt (Grade answer correctness) Output: result.json ({"score": 1.0, "reasoning": "The answer is correct."})