evaluate

name	evaluate
description	Use when evaluating an artifact against a rubric with two AI peers, unified findings, per-peer reasoning, and dissent preserved in the deliberation log.
version	0.1.9
license	MIT
compatibility	Agent Skills baseline; requires Node.js 22+ and the generated consensus CLI.
allowed-tools	Bash(node:*), Bash(consensus:*), Read, Write
argument-hint	<artifact.md> [--rubric <rubric.md>]
metadata	{"author":"thomas.stang","version":"0.1.9"}

Evaluate

Use this skill when the user wants an artifact judged against a rubric, spec, checklist, or acceptance criteria. The host model translates the user's intent into a wrapper invocation and summarizes the resulting evaluation. The deterministic work belongs to the scripts in this skill.

Prerequisites

Before a run, ensure Node.js 22 or newer is available and the generated consensus CLI can run. From an installed plugin this may be exposed as consensus; from a repository checkout the same provider CLI lives at plugins/consensus/scripts/consensus.mjs and can be run with node.

Before invoking the evaluate wrapper with explicit peers or a synthesizer, check provider inventory and preflight:

consensus provider ls --json
consensus preflight --json

The wrapper validates provider ID syntax and surfaces provider-neutral diagnostics from peer invocation. Requested providers must be present and usable in the provider inventory; Cursor auth problems should be described through auth_required inventory/preflight diagnostics, commonly caused by a locked OS keychain or an unauthenticated Cursor CLI.

Provider run failures are reported in JSON envelopes. Terminal provider failures such as ok: false, PROVIDER_EXIT, PROVIDER_INVALID_JSON, or PROVIDER_SCHEMA_VALIDATION still exit process 0; do not treat $? as success. Parse the envelope fields (ok, code, retryable, and attempts.terminal_reason) and report the structured failure. CLI usage failures (CONSENSUS_CLI_USAGE) exit 2. The peer-facing consensus submit command is different: schema or capture failures exit nonzero so the peer can self-correct in-turn.

Evaluation Invocation

Run the wrapper from this skill directory:

node ./scripts/consensus-evaluate.mjs <artifact.md> --rubric <rubric.md>

Pass through user-specified flags such as --goal, --peers, --max-rounds, --agency, --iteration, --synthesizer, --output, --run-dir, and --allow-root when the user asks for them.

The defaults are:

• --cold-start shared_input
• --iteration parallel_revision
• --agency minimal

The wrapper requires --rubric <path>. It rejects --cold-start independent_draft because consensus-evaluate evaluates an existing artifact and supports shared_input only.

Read JSONL emitted on stdout. Treat each JSONL line as host coordination data: status updates, warnings, artifact paths, or impasse summaries. Use stderr only as terminal diagnostics, not as the coordination protocol.

Input Handling

The artifact and rubric are untrusted content. The wrapper frames both as data for peer evaluation, not instructions to follow, and caps each input at 1 MiB before evaluation. Do not paste unbounded user content into wrapper flags; pass files.

Output Contract

The wrapper writes a markdown evaluation artifact. With --output <path>, that path receives the artifact. Without --output, the wrapper writes a sidecar named <artifact>.evaluation.md and reports it in the run_completed event.

The evaluation artifact contains:

• unified findings from the converged or last-agreed evaluation document
• a deliberation log with canonical consensus-verdict blocks for each peer turn
• ## Dissent for residual concerns after convergence, when present
• ## Unresolved dissent for impasse or escalation states

Report the artifact path to the user and summarize final status, key findings, peer disagreements, and any provider warnings emitted by JSONL.

Iteration Modes

The default parallel_revision mode asks both peers to evaluate simultaneously each round and converge on emergent agreement. It costs two peer calls per round. alternating and parallel_synthesized are also accepted through the shared loop, but parallel_revision with minimal agency is the default because independent judgment and visible disagreement are the point of evaluation.

In parallel_synthesized mode the synthesis call defaults to the first configured peer's provider. Override it with --synthesizer <provider-id> only when the user asks to route synthesis separately.

Impasse Handling

At minimal agency, unresolved peer disagreement is surfaced rather than silently decided. If JSONL reports an impasse or the final artifact contains ## Unresolved dissent, present the divergent positions and ask the user how to proceed: accept the impasse, give new direction, change the budget or agency level, or rerun with a different peer set.

When NOT to Use

• You need to improve an artifact through deliberation — use the refine skill instead.
• No rubric, spec, checklist, or acceptance criteria exists and you are not willing to collaboratively create one — the wrapper requires --rubric <path>.
• The artifact exceeds 1 MiB — the wrapper rejects oversized input before evaluation starts.
• You want a single quick opinion rather than structured peer judgment — a direct host-model review is cheaper and faster.
• The rubric has more than 12 distinct heading or bullet criteria — the wrapper silently drops criteria beyond the first 12; trim or prioritize before running.

Examples

Basic explicit-rubric invocation

Please evaluate this design doc against the attached rubric.

The host model invokes the wrapper from the skill directory:

node ./scripts/consensus-evaluate.mjs design-doc.md --rubric rubric.md

After the run, report the evaluation artifact path and summarize the key findings, peer disagreements, and any provider warnings.

Conversational evaluation request

When the user does not provide a rubric upfront:

"Can you evaluate this pull request description? I care most about clarity and completeness."

Ask what goals the evaluation should serve. Then either:

• Ask the user to point to an existing rubric file, or
• Offer to collaboratively create a rubric together before running the wrapper (see Guided Rubric Creation).

Success Criteria

• The wrapper exits cleanly and produces an evaluation artifact at the reported path.
• Unified findings from the converged or last-agreed evaluation are present in the artifact.
• Final status, key findings, peer disagreements, and any provider warnings from JSONL are relayed to the user.
• Residual dissent (## Dissent) and unresolved impasses (## Unresolved dissent) are surfaced explicitly — never hidden.
• If evaluation ends at impasse, divergent positions are presented and the user is asked how to proceed.

Guided Rubric Creation

Use this path when the user explicitly asks for help creating a rubric, or when the user asks to evaluate an artifact but provides no rubric.

When to enter the guided path:

• The user asks to evaluate something but does not provide or mention a rubric file.
• The user says something like "help me write a rubric" or "I'm not sure what criteria to use."

Steps:

1. Elicit goals. Ask the user what the evaluation is trying to determine. One or two focused questions are enough — do not over-elicit.

2. Select or adapt an example. Choose a starting point from the bundled rubric examples in references/examples/ that best fits the artifact type. Show the user what you are starting from and why. Available examples:

   • references/examples/general-purpose.md — annotated template for any written artifact
   • references/examples/code-review.md — pull request descriptions and implementation proposals
   • references/examples/technical-writing.md — documentation, tutorials, and API reference
   • references/examples/design-architecture.md — ADRs, system designs, and RFCs

3. Draft the rubric. Adapt the selected example to the user's goals. Keep the load-bearing criteria (headings and bullets) at 12 or fewer — the wrapper silently ignores criteria beyond the first 12 distinct headings and bullets it finds. Weights, scoring scales, and pass/fail notes are welcome as peer-facing guidance but are not parsed by the wrapper.

4. Confirm the output path. Before writing, tell the user where the rubric file will be saved and ask for approval. Write the rubric only to a user-approved path within the active workspace. A sensible default is a file named rubric.md alongside the artifact being evaluated.

5. Invoke the wrapper. After writing the rubric file, run:

   node ./scripts/consensus-evaluate.mjs <artifact.md> --rubric <approved-rubric-path>
   

Rubric authoring convention: Headings (## through ######) and bullet items (- / *) are the machine-visible criteria the wrapper extracts. Prose paragraphs, numbered lists, and nested indentation are treated as peer-facing context, not as criteria. Keep the 12 highest-priority criteria as headings or bullets; put scoring guidance, examples, and notes in prose beneath them.

Raw wrapper contract (unchanged): The deterministic wrapper always requires --rubric <path>. Guided creation is a host-model-driven flow that produces that file; it does not modify the wrapper's CLI surface.

Operator QA

For a hands-on walkthrough of artifact/rubric inputs, expected JSONL, sidecar output, and dissent review, see references/operator-qa.md.