cli-evals

name	cli-evals
description	Read this skill when working on the planner CLI's eval harness (apps/cli/evals) - the SDK-integration eval, the planner-step evals, the artifact bootstrap - or when adding/capturing a new eval case. Covers the two eval layers, how a case is structured, how to run each eval, and how to author a new case + rubrics.
disable-model-invocation	true

CLI eval harness (apps/cli/evals)

Two eval layers over one shared framework and one per-repo corpus. Both run the real thing as a subprocess (claude -p for the SDK agent; the planner CLI for planner steps) rather than mocking.

• Layer 2 - SDK-integration eval (sdk-integration/): checks out a client repo at a chosen sha (which has the client's real "golden" SDK integration), strips the integration to a clean tree, drives claude -p (Bedrock) to re-implement it, and an agentic judge (OpenRouter, direct) compares the agent's diff to golden. This is the primary eval.
• Layer 1 - planner-step evals (planner/): re-runs one planner step (kb, entityAudit, scenarioRecipe) fresh on the clean tree and grades the produced artifact against an authored findings rubric with a single-shot judge.
• Bootstrap (planner/bootstrap.ts): runs the planner steps through scenarioRecipe on the clean tree to generate a case's frozen artifacts/ (the spec both layers consume). recipeBuilder is not bootstrapped - it hands off to the SDK-integration agent, which generates recipe.json at eval time, so a case carries no frozen recipe.

Directory map

apps/cli/evals/
  framework/          # shared, public: checkout, drive-claude, run-planner-step, corpus, paths,
                      #   bedrock, env, git, judge-model
  sdk-integration/    # Layer 2: run.ts, judge.ts (agentic), verdict.ts, case.ts
  planner/            # Layer 1: run.ts, bootstrap.ts, steps.ts, judge.ts (single-shot), README
  cases/<repo>/       # the corpus - one folder per client repo (OPENSOURCE-IGNORED: client IP)
  .cache/  .runs/     # gitignored: repo cache, per-run outputs

A case (cases/<repo>/)

file	what	required for
input.json	{ owner, repo, sha, installationId } - target coords, no source. Optional contextRepos: [{ owner, repo, sha, installationId }] for a multi-repo app (see below)	both
strip.patch	git diff sha->clean: the client's integration removed (manually), so the tree still boots	Layer 2
context-strips/<contextRepo>.patch	multi-repo only: the same sha->clean strip for a context repo, so no staged tree references autonoma (see below). Omit when the pinned context sha predates the integration	both (multi-repo)
artifacts/	frozen planner spec: project-map.json, pages.json, AUTONOMA.md, entity-audit.md, scenarios.md (no recipe.json - the SDK agent generates that at eval time)	both
context.json	{ description, testingGoal, criticalFlows } - written record of the app; nothing reads it since the planner's opening Q&A was removed	none
rubrics/<step>.md	findings rubric per gradable step	Layer 1
agent-notes.md	optional per-case instructions appended verbatim to the driven agent's drive prompt (repo-specific seeding conventions); most cases have none	Layer 2
ENV.md	dev-only, advisory local-boot notes (read by NO harness code)	humans

Derived at run time: clean = sha + strip.patch; golden = the sha checkout as-is; agent = the sandbox after the drive (its git diff is the agent's integration).

Multi-repo (polyrepo) apps

Some apps span several repos - the SDK integration lands in one (the target, named in input.json), but the planner and agent must read the others too. List those as contextRepos (SHAs pinned). The case does NOT label any repo frontend/backend; the planner's mapper discovers roles itself. The two layers just stage them:

• Layer 2 (sdk-integration): the sandbox is the target only; context repos are cloned, staged read-only outside the sandbox, and handed to the drive as extra --add-dirs. Golden and the judge apply to the target only - context repos are never graded.
• Layer 1 (bootstrap / planner): the harness assembles a combined project root with every repo as a sibling subdir (named by its bare repo name) and applies strip.patch to the target subdir (see preparePlannerProject in framework/). The mapper then resolves the scope on its own; pass --frontend <repo> --backends <repo,...> only to override an ambiguous mapping.

Context repos are stripped too, to the same bar as the target. For a realistic eval NO staged tree may reference autonoma - a leftover SDK import, endpoint, CI workflow, or host allowlist would tip the agent off. Each context repo is staged through stageContextCheckout (framework/context-repos.ts), which applies cases/<repo>/context-strips/<contextRepo>.patch (keyed by bare repo name) if present. When the pinned context sha already predates the integration (e.g. eon's eon-rn-web@<sha> - a clean tree, git grep autonoma empty), no patch is needed. NOTE: stripping only cleans the working tree; git history in every checkout still contains the integration - see the caveat below.

Running

# env: GITHUB_APP_PRIVATE_KEY_FILE (a .pem path), OPENROUTER_API_KEY, AWS_BEARER_TOKEN_BEDROCK,
#      AUTONOMA_API_TOKEN (or AUTONOMA_API_KEY). App id + .env loading are handled (see below).

pnpm --filter @autonoma-ai/planner eval:sdk       -- --repo <name>          # Layer 2 (drive + judge)
pnpm --filter @autonoma-ai/planner eval:planner   -- --repo <name> --step entityAudit [--promote]
pnpm --filter @autonoma-ai/planner eval:bootstrap -- --repo <name> [--frontend <app-dir>] [--backends <a,b>]
#   --frontend/--backends are optional: the mapper resolves scope itself, and only needs them to
#   disambiguate (e.g. multiple candidate frontends). For a multi-repo case they name subdirs of the
#   combined root (bare repo names). Omit them and let the mapper decide when the mapping is clear.

Layer 2 flags: --no-drive (checkout+strip only, cheap), --no-judge, --model, --judge-model, --timeout. Outputs land in .runs/<repo>/<stamp>/ (agent.diff, golden.diff, transcript, verdict.json). Run Layer 2 with any host network sandbox disabled - the Bedrock token 403s through a proxy otherwise.

Adding a new case

1. Pick sha - a commit that has the client's integration and boots. Confirm it's on the remote.
2. Capture strip.patch - check out sha, manually delete the SDK integration and its reference sites (route registration, config/env refs) so the stripped tree still builds, then git add -A && git diff --cached > cases/<repo>/strip.patch; reset the repo. Verify it applies to the remote sha (eval:sdk --no-drive does this).
3. Write input.json (coords + real installationId). A context.json is optional and read by nothing - write one only as a record of what the app is and which flows matter. For a multi-repo case, list the siblings in contextRepos, then for each one confirm the staged tree is autonoma-free: git grep -i autonoma <contextSha>. If it has any hit, capture a context-strips/<contextRepo>.patch the same way as step 2; if it is already clean, add nothing.
4. Generate artifacts/: eval:bootstrap --repo <name> --frontend <app-dir>.
5. Author rubrics/<step>.md - a findings list a correct artifact MUST contain, grounded in the repo. Ground the entityAudit rubric in real creation functions; a good source is a passing SDK run (its factories prove which creation paths are real). These are a regression floor, not an oracle.
6. Optionally add ENV.md (dev boot notes) and run eval:sdk --repo <name> end to end.

Key design decisions (don't undo these without reason)

• No machine-read boot config. Getting the app running locally is the driven agent's own best-effort job (it discovers the stack + start command). ENV.md is human-only docs.
• The judge calls OpenRouter directly (framework/judge-model.ts, OPENROUTER_API_KEY), NOT the CLI's /v1/llm-proxy credit proxy - it's harness code, not a billed customer. The planner subprocess necessarily uses the proxy (that's the only mode the product has).
• No @autonoma/* workspace deps in the CLI. The judges use the CLI's own runAgent + tools, not @autonoma/ai, to keep the published package's graph clean (and avoid the sharp pull).
• The GitHub App id is a committed default (DEFAULT_GITHUB_APP_ID, non-secret); only the private key is a secret, read from GITHUB_APP_PRIVATE_KEY_FILE (a multiline PEM breaks --env-file). The eval scripts load the repo-root .env via --env-file.
• cases/ is committed but opensource-ignored (client IP: strip patches + artifacts). Client repo source is never committed - it's fetched by sha into the gitignored .cache/.
• Bedrock for the drive, single artifact set per case (no variant system).

Gotchas

• A planner step can fail transiently on a weak/flaky model (empty provider error on gemini-3-flash-preview); eval:planner reports it cleanly and you re-run, or pass --model.
• The judge's default model id (anthropic/claude-opus-4.8) must resolve on OpenRouter; override with JUDGE_MODEL. Its 1M context is load-bearing for the SDK-integration judge - that judge greps and reads its way around a whole client repo over up to 40 turns, and on a large monorepo the accumulated tool output overran a 200K window and killed the run with no verdict, after a paid drive.
• Layer 1 seeds a step's upstream artifacts from artifacts/ but never the step's own output, so a step never grades against its own frozen answer.
• strip.patch / context strips only clean the working tree, not git history. Every staged checkout is a full clone, so the golden integration is still reachable via git log/git show (git log -p | grep autonoma finds it). A grep autonoma over the files comes back empty, but an agent that inspects history can still recover the answer. Making the sandbox history-clean (re-init git after the strip so the clean baseline is the root commit) is a separate, harness-wide change.