j-rig-eval

# JRig Skill Evaluation ## Overview Evaluate an Agent Skill with JRig's binary-evaluation harness, returning the rollout decision first and citing the evidence behind failures or uncertainty. Treat observed behavior as authoritative over claims in `SKILL.md`. JRig evaluates five dimensions by default. The opt-in regression and naked-model checks need additional inputs and can roughly double cost. Always read the rollout decision; a completed run may still warn, block, or require review. ## Prerequisites Before running an evaluation, confirm: 1. Node.js 20 or newer and `j-rig` are installed. 2. The target directory contains the `SKILL.md` to evaluate. 3. An evaluation spec is available. Pass `--spec` unless the target root contains `eval-spec.yaml` or `eval-spec.yml`. 4. Credentials for a real supported provider are present. Stub mode is useful for plumbing tests only and is not release evidence. 5. Any command named by the spec is trusted. `--run-self-test` executes that command without a shell and with a scoped environment, but it still runs target-controlled code. ## Instructions ### Step 1: Resolve inputs Identify the target skill directory, evaluation spec, provider, and models. Read the spec and note blocker criteria, sacred regression cases, self-test commands, and expected evidence paths. If the user supplies `--full`, translate it into both opt-in checks: - add `--regression-baseline BASELINE_JSON`; - add `--baseline-check`. Ask for the regression baseline path if it is not already known. Do not invent one. ### Step 2: Run the preflight ```bash j-rig check TARGET_SKILL --json ``` Stop and repair package-integrity failures before evaluation. The eval command then validates its spec and provider configuration. Never report stub-backed results as a production gate. ### Step 3: Run the evaluation Prefer machine-readable output and an evidence bundle: ```bash j-rig eval TARGET_SKILL \ --spec EVAL_SPEC \ --models MODEL_ID \ --provider PROVIDER_ID \ --json \ --emit-bundle EVIDENCE_JSON ``` Add only necessary controls: `--run-self-test`, `--samples 3`, judge provider and model flags, regression or baseline checks, `--trace-boundary`, or the explicit scope reductions `--no-trigger` and `--no-functional`. The repository's dogfood spec is `skill/eval.yaml`, so evaluate it with an explicit `--spec skill/eval.yaml`. ### Step 4: Interpret the result Read the JSON rollout decision rather than treating exit code zero as approval: - `ship`: all evaluated blockers and sacred cases passed; - `warn`: evaluation completed with non-blocking failures or uncertainty; - `block`: a blocker or sacred regression failed; - `obsolete_review`: the naked model matched the skill and the skill's value should be reviewed. Exit code `0` means evaluation completed, including completed `warn`, `block`, and `obsolete_review` results. Exit code `1` means a package or runtime failure. Exit code `2` means the judge path failed or nothing could be evaluated; retain the generated error evidence and do not manufacture a rollout decision. ### Step 5: Report evidence Lead with the exact decision and models. Name failed, uncertain, blocker, and sacred criteria; included and omitted dimensions; non-secret provider and judge identities; artifact paths; and the smallest remediation and rerun command. ## Output Return a concise evidence-backed gate report: ```text Decision: BLOCK Models: MODEL_IDS Coverage: default 5/7; regression included; baseline omitted Blocking evidence: - CRITERION: observed failure and evidence pointer Artifacts: JSON, database, and bundle paths Next action: specific repair and rerun command ``` The `--emit-bundle` artifact is a kernel-validated array of in-toto `gate-result/v1` statements, one per model. Preserve it with the release record. ## Error Handling - Preflight failure: correct the target package and rerun `j-rig check`. - Eval setup failure: correct the spec or provider configuration and rerun. - Missing default spec: pass `--spec`; only two default filenames are discovered. - Provider or judge failure: retain evidence, fix it, and never substitute stubs. - Exit code `2`: report that no valid gate decision was produced. - Self-test failure: inspect the trusted spec command and its scoped inputs; never broaden execution permissions to make it pass. - `warn` or `block`: remediate the named criteria and rerun the same model and option set so results remain comparable. ## Examples Evaluate the repository dogfood skill with default five-dimension coverage: ```bash j-rig check skill --json j-rig eval skill --spec skill/eval.yaml --models MODEL_ID \ --provider PROVIDER_ID --json --emit-bundle artifacts/j-rig-evidence.json ``` Run all seven dimensions when a valid regression baseline is available: ```bash j-rig eval TARGET_SKILL --spec EVAL_SPEC --models MODEL_ID \ --provider PROVIDER_ID --regression-baseline BASELINE_JSON \ --baseline-check --json --emit-bundle EVIDENCE_JSON ``` ## Resources - [Evaluation contract and decision semantics](references/evaluation-contract.md) - [Repository README](https://github.com/jeremylongshore/j-rig-skill-binary-eval#readme) - [Evaluation specification example](eval.yaml) - [Apache 2.0 license](https://github.com/jeremylongshore/j-rig-skill-binary-eval/blob/main/LICENSE)