القائمة
Add a new evaluation judge type so task packs can use it to verify agent correctness. Use when creating a new check, custom validator, or specialized assertion.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Plan and execute code changes using a spec-driven workflow with structured documents. Use when making non-trivial changes (multi-file, multi-step, or behavior-impacting) that need structured planning before implementation. Not for trivial single-file edits.
Create and manage changesets for versioning with @changesets/cli. Use when making changes that need a changelog entry, preparing a release, or managing package versions.
Diagnose and fix failed benchmark runs. Use when a benchmark run fails, scores are unexpected, judges show errors, or traces are incomplete.
Build the pnpm monorepo correctly respecting inter-package dependencies. Use when building, verifying changes, or troubleshooting build failures.
Write and run tests using Node.js built-in test runner (node:test) with .mjs files. Use when creating new tests, debugging test failures, or understanding test infrastructure.
Understand the vanilla JS web-report app structure, patterns, and contribution guidelines. Use when modifying the UI, adding new views, or fixing frontend bugs.
| name | add-judge |
| description | Add a new evaluation judge type so task packs can use it to verify agent correctness. Use when creating a new check, custom validator, or specialized assertion. |
Add a new judge type to the evaluation system.
command, test-result, lint-check, file-exists, file-contains, json-value, json-schema, glob, file-count, snapshot, patch-validation, token-efficiency) don't cover the evaluation need.Define the type in packages/core/src/types.ts:
interface XxxJudge with id, label, type: "xxx", and judge-specific fields.TaskJudge union type.Implement in packages/judges/src/index.ts:
runXxxJudge() returning Promise<JudgeResult>.judgeId, label, type, exitCode, success, stdout, stderr, durationMs, critical: judge.critical ?? false.passedCount, failedCount, totalCount, warningCount, errorCount.resolveJudgeWorkingDirectory(), buildStepEnvironment(), executeCommand() for consistent execution.runJudge() dispatch switch.Add loader support in packages/taskpacks/src/index.ts:
normalizeJudge(), add parsing for type === "xxx".assertString(), assertOptionalPositiveInteger(), etc."xxx" to the supportedTypes array in the error message.If the judge contributes to scoring, add a scoring helper in packages/report/src/scoring.ts and wire it into the relevant weight modes.
Verify:
pnpm --filter @agentarena/judges buildpnpm --filter @agentarena/taskpacks buildtests/judges.test.mjs.runJudge() — defined but never called is a bug.critical field defaults to false consistently with all other judges.