Run isolated eval and grading calls using CC 2.1.81 --bare mode. Constructs claude -p --bare invocations for skill evaluation, trigger testing, and LLM grading without plugin/hook interference. Use when running eval pipelines, grading skill outputs, benchmarking prompt quality, or testing trigger accuracy in isolation.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Run isolated eval and grading calls using CC 2.1.81 --bare mode. Constructs claude -p --bare invocations for skill evaluation, trigger testing, and LLM grading without plugin/hook interference. Use when running eval pipelines, grading skill outputs, benchmarking prompt quality, or testing trigger accuracy in isolation.
Run claude -p --bare for fast, clean eval/grading without plugin overhead.
CC 2.1.81 required. The --bare flag skips hooks, LSP, plugin sync, and skill directory walks.
When to Use
Grading skill outputs against assertions
Trigger classification (which skill matches a prompt)
Description optimization iterations
Any scripted -p call that doesn't need plugins
When NOT to Use
Testing skill routing (needs --plugin-dir)
Testing agent orchestration (needs full plugin context)
Interactive sessions
Prerequisites
# --bare requires ANTHROPIC_API_KEY (OAuth/keychain disabled)export ANTHROPIC_API_KEY="sk-ant-..."# Verify CC version
claude --version # Must be >= 2.1.81
Quick Reference
Call Type
Command Pattern
Grading
claude -p "$prompt" --bare --max-turns 1 --output-format text
Trigger
claude -p "$prompt" --bare --json-schema "$schema" --output-format json
Streaming grade
claude -p "$prompt" --bare --max-turns 1 --output-format stream-json
Optimize
echo "$prompt" | claude -p --bare --max-turns 1 --output-format text
Force-skill
claude -p "$prompt" --bare --print --append-system-prompt "$content"
@-file in prompt
claude -p "grade @fixtures/case-1.md against rubric" --bare (CC 2.1.113 Remote Control autocomplete)
Long harness runs (CC 2.1.199+): set CLAUDE_CODE_RETRY_WATCHDOG=1 for unattended eval batches — it raises the default retry count for non-capacity transient errors to 300 and lifts the cap of 15 on CLAUDE_CODE_MAX_RETRIES, so an overnight grading run survives transient API blips instead of dying mid-batch.
--output-format stream-json
Newline-delimited JSON events (one per token/tool-call) — lets a runner score partial output or abort early on a failing probe without waiting for the full response.
claude -p "$prompt" --bare --max-turns 1 --output-format stream-json \
| while IFS= read -r line; do# line is a single JSON event; inspect $.type == "content_block_delta"
jq -r 'select(.type == "content_block_delta") | .delta.text' <<< "$line"done
Use stream-json over json when:
grading long outputs and you want incremental scoring,
piping into another CLI step-by-step (e.g. ork:eval-runner),
you need per-token timing data alongside the content.
OrchestKit's eval scripts (npm run eval:skill) auto-detect bare mode:
# eval-common.sh detects ANTHROPIC_API_KEY → sets BARE_MODE=true# Scripts add --bare to all non-plugin calls automatically
Bare calls: Trigger classification, force-skill, baseline, all grading.
Never bare:run_with_skill (needs plugin context for routing tests).
CC 2.1.119: --print honors agent tools: / disallowedTools: (M122)
Before CC 2.1.119, --print mode ran with the full default tool set regardless of the agent's frontmatter tools: and disallowedTools:. Bare-eval grading was effectively ungated — graders could call any tool they wanted, even if the agent definition restricted them.
As of 2.1.119, --print enforces the agent's declared tool surface. Implications for eval design:
Consequence
Action
Eval graders that relied on unrestricted tool access may now fail
Audit grader prompts for tools they actually need; whitelist explicitly via the agent's tools: frontmatter
Eval results match interactive runs
Reproducibility improves — grading what the model can actually do, not what it could do in an unsandboxed --print
--agent <name> also honors permissionMode in --print
Permission-gated tools (Bash, Edit) require either permissionMode: acceptEdits or explicit allowlists in the agent definition
Migration test:
# Run an eval against an agent with a deliberately tight tools: list.# Graders that previously called Read/Bash freely will now fail unless those# tools are declared on the agent.
claude -p "$prompt" --bare --print --agent grader-test
If the grader fails with a "tool not permitted" error, add the required tool to the agent's tools: frontmatter and re-run.
CC 2.1.121: CLAUDE_CODE_FORK_SUBAGENT=1 for grader determinism (#1545)
Before CC 2.1.121, the env var only worked in interactive sessions. As of 2.1.121, non-interactive paths (claude -p, SDK) honor it too — each grader invocation gets a fresh forked subagent context.
The cross-eval state-leak problem this fixes:
Without forking, sequential claude -p --bare graders inherit harness state:
Inherited
Symptom
memory MCP query cache
grader sees stale hit from previous run; same fixture grades differently
.claude/chain/*.json on disk
grader for "implement" thinks "explore" already ran (file is from previous test)
ToolSearch deferred-tool cache
first grader's MCP loads bleed into next grader's tool registry
model picker pref
grader N inherits --model=opus from grader N-1
This produced ~5–10% retry rate and non-reproducible scores — the eval baseline drifted between runs, engineers chased phantom regressions.
Fix:tests/evals/scripts/lib/eval-common.sh exports CLAUDE_CODE_FORK_SUBAGENT=1, so every script that sources it (run-trigger-eval, run-quality-eval, run-agent-eval, optimize-description, etc.) gets forked graders automatically. The CI workflow .github/workflows/skill-eval.yml also sets it at the workflow level (its predecessor orchestkit-eval.yml was retired 2026-08-01 — its grading phases rated description prose, not behavior). Older CC silently ignores the env var (no-op).
Determinism contract: running the same grader on the same fixture twice in a row produces the same score. Verified by tests/evals/scripts/test-grader-determinism.sh.
workflows/skill-fitness.js is a runnable dynamic-workflow template — the workflow-backed complement to the static conformance grader (scripts/eval/conformance-check.mjs). It fans out one isolated-context agent per skill to score fitness (freshness / router-clarity / structure) and synthesizes a ranked scorecard, catching qualitative drift a static grep can't (description/body count mismatches, duplicate headings, install-specific absolute paths, version drift). Run it with the Workflow tool:
Treat it as a template, not a verbatim script — adapt the SKILLS list and rubric per use. Cost is real (~50k tokens/skill; scoring all ~112 is ~6M tokens), so pass an explicit batch via args. Static-first: run conformance-check.mjs (zero tokens) to pre-filter, then this harness for the judgment grep can't make.
Holdout Bake-Off Grading
The holdout-promotion gate grades a champion and a challenger SKILL.md over the same sealed holdout via bare-mode forked graders — the canonical consumer of the determinism contract above: identical grader + identical ork-rubric/1.0 + identical sealed set, with CLAUDE_CODE_FORK_SUBAGENT=1 so the only variable is the version under test. Both --bare constraints apply (requires ANTHROPIC_API_KEY, bills tokens directly → on-demand / CI only). Run it with bash tests/evals/scripts/run-skill-eval.sh --holdout-promote <skill>.
