cli-evals

name	cli-evals
description	Read this skill when working on the planner CLI's eval harness (apps/cli/evals) - the SDK-integration eval, the planner-step evals, the artifact bootstrap - or when adding/capturing a new eval case. Covers the two eval layers, how a case is structured, how to run each eval, and how to author a new case + rubrics.
disable-model-invocation	true

CLI eval harness (apps/cli/evals)

Two eval layers over one shared framework and one per-repo corpus. Both run the real thing as a subprocess (claude -p for the SDK agent; the planner CLI for planner steps) rather than mocking.

• Layer 2 - SDK-integration eval (sdk-integration/): checks out a client repo at a chosen sha (which has the client's real "golden" SDK integration), strips the integration to a clean tree, drives claude -p (Bedrock) to re-implement it, and an agentic judge (OpenRouter, direct) compares the agent's diff to golden. This is the primary eval.
• Layer 1 - planner-step evals (planner/): re-runs one planner step (kb, entityAudit, scenarioRecipe) fresh on the clean tree and grades the produced artifact against an authored findings rubric with a single-shot judge.
• Bootstrap (planner/bootstrap.ts): runs the planner steps through scenarioRecipe on the clean tree to generate a case's frozen artifacts/ (the spec both layers consume). recipeBuilder is not bootstrapped - it hands off to the SDK-integration agent, which generates at eval time, so a case carries no frozen recipe.