cautilus-agent

name	cautilus-agent
description	Use when intentful behavior evaluation itself is the task and the repo should run Cautilus's checked-in workflow instead of reconstructing compare, held-out, and review commands by hand.

Cautilus Agent

Use Cautilus Agent when intentful behavior evaluation itself is the task and the repo wants to run the checked-in Cautilus workflow instead of rebuilding eval fixtures, packets, reports, claim discovery, review, or improve commands by hand. For external host repos, treat evaluate fixture, evaluate observation, and post-run eval skill-experiment compare as the stable cross-repo surface; claim discovery automation, improve automation, live app-runner workflows, and review-learning capture stay opt-in. eval skill-experiment compare compares host-preserved baseline and variant outputs; it does not clone, install, or execute skills.

Cautilus Agent assumes a Cautilus binary is available. In the Cautilus product repo itself, prefer the checked-in source launcher ./bin/cautilus over cautilus on PATH, because the installed machine binary can lag the current checkout. In consumer repos, use cautilus on PATH. If no binary is available, install the CLI first and verify with cautilus --version. To materialize Cautilus Agent in a host repo, run cautilus init --repo-root ..

Product Shape

Cautilus is a standalone binary plus Cautilus Agent. Host repos own adapters, fixtures, prompts, wrappers, and policy. The binary owns command discovery, packet examples, deterministic scans, validation, and reusable evaluation artifacts. Cautilus Agent owns routing, sequencing, user-facing decision boundaries, and LLM-backed claim review work. eval and improve may still exercise model-involving behavior through adapter-owned runners.

The current external-adoption front door is eval: verify bounded intentful behavior with explicit fixtures and adapters, and compare host-preserved skill experiment outputs after a run. The broader product also includes opt-in claim and improve surfaces. Do not present claim, improve, live app-runner workflows, or review-learning packet capture as stable cross-repo defaults; they are opt-in surfaces a host repo adopts deliberately.

CLI First

Resolve the binary before workflow commands:

CAUTILUS_BIN=cautilus
if [ -x ./bin/cautilus ]; then CAUTILUS_BIN=./bin/cautilus; fi

Let the binary print command families and packet examples. Use "$CAUTILUS_BIN" doctor commands --format json, "$CAUTILUS_BIN" --help, portable cautilus doctor commands --format json / cautilus --help, or a command's --example-input / --example-output surface instead of copying broad command lists into the answer; structured stdout defaults to YAML, so use --format json when a parser needs JSON. Use command-cookbook.md only after the binary has identified the relevant command family and a concrete multi-step invocation is needed.

No-Input Orientation

When invoked with no task detail, orient first:

"$CAUTILUS_BIN" doctor status --repo-root . --format json

Read cautilus.agent_status.v1 as the current product map. Summarize binary health, agent-surface readiness, adapter state, selected claim-state availability, scan entries, linked docs depth, and nextBranches. When claimState.orientationState is present, treat it as the selected claim map for status and branch commands. Keep claimState.configuredState as the writable discovery baseline, not necessarily the most useful review packet. Then help the user pick the next branch or stop, presenting branch labels and reasons in coordinator-facing language first. Treat each nextBranches[].label as the human choice name; branch id is a secondary stable packet reference only when it helps auditability. If the user answers with a number or short label after you present branch choices, resolve that answer against the branch labels you just showed and execute that selected branch. Do not reinterpret a numbered branch selection as a request for another status summary.

If nextBranches includes initialize_adapter and the user delegated setup continuation, run the adapter setup branch and then rerun doctor status. If claim state is missing, present the bounded scan entries and depth, explicitly ask the user to "confirm this scan scope or adjust it", state that LLM review needs a separate review budget, and only then enter claim discovery. If claim state exists, read or refresh that packet before planning new proof work.

Declared Claim Discovery

Use this path when the user asks whether a repo proves what it claims, whether docs and behavior are aligned, or which scenarios still need to be created. In external consumer repos, use this path only after the user explicitly opts into non-eval Cautilus work. For these direct questions, do not run discover claims until scan entries/depth are stated and the user confirms or adjusts the scope; keep LLM review as a separate budgeted branch. Do not hard-code the search to README; by default, the binary starts from adapter-owned claim_discovery.entries or README.md/AGENTS.md/CLAUDE.md and follows repo-local Markdown links to depth 3. Use repeated --source arguments only when the user or adapter has selected an explicit truth-surface inventory. Before the first broad scan, check the entry docs for non-claim section conventions and for non-English or repo-specific claim vocabulary, propose claim_discovery.classification_hints (non_claim_section_headings, claim_lexicon_terms) with before/after extraction counts for the maintainer to ratify into the adapter — dry-run proposals with discover claims --adapter <proposed.yaml> without writing to the repo; the binary executes ratified hints deterministically.

Initial scan:

"$CAUTILUS_BIN" discover claims --repo-root . --output <claims.json>
cautilus discover claims --repo-root . --output <claims.json>

Refresh from existing state:

"$CAUTILUS_BIN" discover claims --previous <claims.json> --refresh-plan --output <refresh-plan.json>

When the selected branch is "compare the saved claim map with recent repo changes", "refresh existing claim state", or an equivalent refresh_claims_from_diff branch, run the refresh-plan command above and read refreshSummary. Do not satisfy that branch by running only discover claims status; discover claims status belongs to the separate "inspect the saved claim map" branch. Read refreshSummary first. Explain it as a saved claim map catching up to current repo changes, not as a completed claim refresh. Say what was recorded, what was not changed yet, and which next choice is safest. Prefer labels such as "compare the saved claim map with recent repo changes" and "update the saved claim map before review or eval planning"; include internal branch ids only as references when needed. After writing a refresh plan, inspect .refreshSummary directly, for example with jq '.refreshSummary' <refresh-plan.json>. Do not reconstruct changedClaimCount, carriedForwardClaimCount, or source hotspots from raw changedSources or claimPlan when refreshSummary exists.

Status from existing state:

"$CAUTILUS_BIN" discover claims status --input <claims.json> --sample-claims 10

Before creating fixtures, keep proof class and readiness separate. Use human-auditable for source/doc judgment, deterministic for unit/lint/type/build/schema/CI proof, cautilus-eval for model/agent/prompt/skill/workflow evidence, needs-scenario for claims needing scenario decomposition, and needs-alignment for docs/code/adapter/skill surfaces that must be reconciled before proof would be honest.

After discovery or refresh, summarize scanned entry files, linked docs count and depth, raw candidate count, claim summary by proof mechanism/readiness/evidence/review/lifecycle, and the groups that look ready for deterministic tests, Cautilus scenarios, alignment work, or human-auditable review. When the next natural branch is claim review, explain that it is a budgeted LLM review branch before presenting it as a choice. The coordinator should understand that choosing review means setting a review budget before any reviewer lanes, result application, eval planning, edits, or commits happen. Use discover claims status --sample-claims <n> as the canonical status view before hand-inspecting packet fields. Read discoveryBoundary before judging false negatives. If a declared promise is present inside the configured entry docs or linked Markdown graph and discovery missed it, report a discover claims false-negative bug; if a user-facing feature is missing from that graph, report it as an entry-surface or narrative gap unless another reviewed artifact proves the binary skipped an in-scope declaration. During curation, reduce false positives before proof work and scan for likely missing public promises so the user can decide whether they were intentionally excluded or simply under-documented. Read actionSummary.primaryBuckets before making a next-work recommendation. Use the bucket recommendedActor and summary fields to separate agent work, human confirmation, deterministic proof, Cautilus eval planning, scenario design, alignment, and split-or-defer branches. Use actionSummary.crossCuttingSignals for review debt or stale-evidence warnings that can coexist with a primary proof branch. When preparing a focused review queue, pass --action-bucket <bucket> to discover claims review-input instead of hand-filtering claim JSON. When triaging needs-scenario or needs-alignment, stay inside packet surfaces: prepare the focused action bucket, record reclassification decisions as cautilus.claim_review_result.v1, and apply them with discover claims apply-review. Promote only concrete command, packet, runner, schema, or skill-behavior claims to ready-for-proof; keep product-boundary, future-surface, ownership, umbrella, and readiness-definition claims non-ready until split or aligned. In the Cautilus product repo, when raw claim status or review packets are too large for a maintainer to judge directly, run npm run claims:status-report and read .cautilus/claims/claim-status-report.md before asking for human decisions. If the maintainer is reviewing from a constrained terminal or phone, run npm run claims:status-server so they can read the report in a browser and save section comments as .cautilus/claims/claim-status-comments.json. When raw candidates are too granular for product judgment, curate a canonical claim spec tree before continuing HITL. Treat raw candidates as high-recall proof-planning inputs, not as the human-facing promise map. For Cautilus itself, keep the curated user-facing promises in docs/specs/user/index.spec.md and per-claim pages under docs/specs/user/. Keep the maintainer-facing index in docs/specs/contracts/index.spec.md. In the Cautilus product repo, product-meaning review should start from those spec docs; use the status report for packet audit, debugging, or deciding which remaining raw candidates are not yet absorbed. User-facing claims must use plain product language. Order user-facing spec indexes by the user's feature mental model before cross-cutting implementation promises. For Cautilus itself, lead with claim, eval, improve, then doctor or readiness, then supporting promises such as portability, packet/reporting surfaces, and proof-debt visibility. For other repos, infer the equivalent top user jobs from the adapter, README, and source docs instead of copying Cautilus-specific command names. Use adapter semantic_groups, source-doc headings, declared product surfaces, and README or guide structure as the portable signal for those top user jobs. Keep each index short; put subclaims and evidence placeholders in the per-claim spec pages. Maintainer-facing claims may use internal terms, but they must stay aligned with the user-facing claim specs and preserve source refs, proof route, evidence status, and next action. Review packets and curation artifacts preserve absorbed raw claim ids and fingerprints when available; stable catalog docs may summarize the absorbed raw themes instead of listing volatile line-based ids. When those catalogs change in the Cautilus product repo, run npm run claims:canonical-map before npm run claims:status-report so the report shows how raw user claims compress into the current U-claim catalog. If discover claims status or doctor status reports gitState.isStale=true, run discover claims --previous <claims.json> --refresh-plan before claim review, review-result application, or eval planning. Do not launch reviewers, apply review results, plan evals, edit files, or commit artifacts from a stale claim packet unless the user explicitly asks to override stale state. If a view is missing, prefer adding a product-owned summary option or review packet over guessing raw JSON keys with ad hoc jq. If a new discovery run changes only volatile metadata such as the reviewed git commit and not source inventory, candidate claims, labels, or evidence refs, report it as a semantic no-op and do not create a pointer-only commit.

When reporting a refresh-plan result to a coordinator, use this shape:

• What I did: recorded a comparison between the saved claim map and the current checkout.
• What I found: cite refreshSummary.changedSourceCount, changedClaimCount, carriedForwardClaimCount, and the top changedClaimSources.
• What I did not do: did not update the saved claim map, review claims, plan evals, or edit product files unless the user selected that branch.
• Recommended next choice: use the first safe refreshSummary.nextActions item in plain language.

LLM-backed claim review is a separate branch; before launching it, state the selected review budget in these exact labels: maximum clusters, claims per cluster, parallel lanes, excerpt chars, retry policy, and skipped-cluster policy. Then ask the user to confirm or adjust that budget, and stop until that confirmation or adjustment arrives. If the user selects the review branch without naming a budget, use the command defaults as a conservative deterministic prepare-input budget and say so before running discover claims review-input. After discover claims review-input, restate the concrete selected budget with the same labels before any reviewer lane, even if the user already asked to launch a reviewer. After a fresh first discovery for any review branch, run discover claims status --input <claims.json> --sample-claims <n> before discover claims review-input; this is the product-owned review queue summary, not optional narration. After discover claims review-input, stop at the packet boundary unless the user has explicitly delegated reviewer launch. When reviewer launch is explicitly delegated, use the smallest honest launch budget if none was provided: one cluster, one claim, one reviewer lane, no retries. The default single-lane launch in an agent session is the current agent acting as the reviewer lane: review the selected cluster and write a valid cautilus.claim_review_result.v1 packet. For claim evidence, prefer evidenceStatus=unknown unless the review input includes direct verified evidence; when attaching Cautilus eval evidence, use kind=eval-summary for eval-summary.json, kind=eval-observed for eval-observed.json, and reserve kind=fixture for checked-in fixture or scenario input files. Use an external reviewer CLI helper only when that external lane was explicitly selected. The launch is only complete when the selected reviewer lane returns a cautilus.claim_review_result.v1 packet; if an external lane is blocked from reaching its model provider, report a blocked reviewer launch rather than treating the helper invocation as evidence. This branch proves reviewer launch, not review-result merge behavior. It ends when the selected reviewer lane writes the cautilus.claim_review_result.v1 packet and reports the packet path, reviewed claim count, evidence label decisions, unresolved questions, and next branch choices. If you need local confidence before stopping, inspect the written JSON shape directly; do not call discover claims apply-review as packet validation inside the reviewer-launch branch. Treat discover claims apply-review as the next branch even when the output path is temporary and the intent is only validation. After reviewer launch, stop before review-result application, eval planning, edits, or commits unless the user explicitly delegates the next branch. In the later review-to-eval branch, apply cautilus.claim_review_result.v1, validate the reviewed claim packet, and only then plan eval fixtures for reviewed cautilus-eval claims that are ready-for-proof. When review-to-eval is explicitly delegated in the same agent turn, keep the same bounded reviewer budget unless the user names a larger one, write the applied claim packet to a separate artifact, validate that artifact, run evaluate claims plan from the validated artifact, and stop before fixture authoring, eval execution, product edits, or commits. The review and eval-planning commands reject stale claim packets by default; treat that error as a prompt to refresh, not as a reason to pass --allow-stale-claims automatically.

After a human review, HITL chunk, issue, pull request review, review.json, or review-summary.json produces a source review decision about a Cautilus proposal, use "$CAUTILUS_BIN" review feedback build to materialize cautilus.review_feedback.v1; include proposal evidence for accepted, narrowed, reframed, or rejected, reserve proposal-less packets for missing_critical, read the packet before reporting it, and do not treat it as eval pass/fail or source-ref verification; when comparing multiple explicitly selected review-feedback packets, use "$CAUTILUS_BIN" review feedback summarize --input <packet> ... to materialize cautilus.review_feedback_summary.v1 without inventing active-run packet discovery or default packet placement.

Eval Routing

Cautilus has two top-level evaluation surfaces and four fixture presets. Use dev for AI-assisted development work such as repo contracts, tools, and skills. Use app for AI-powered product behavior such as chat, prompt, and service responses. The canonical command families are claim, eval, and improve. Use cautilus evaluate fixture --fixture <fixture.json> when the repo already has a checked-in fixture and adapter-owned runner. When the agent runtime is read-only, pass an explicit writable --output-dir; prefer /dev/shm/cautilus-<label> when available, otherwise a writable external temp directory. For a fixture-runtime smoke where doctor --scope agent-surface or doctor status already shows the local skill surface is ready, --skip-preflight is the right boundary in a read-only agent runtime; state that preflight was skipped because the ready state was already observed. Routing rule:

• dev / repo: an AI development agent must obey the repo work contract.
• dev / skill: a checked-in or portable development skill must still trigger, execute, and validate cleanly.
• app / chat: multi-turn product conversation behavior regressed after a prompt or wrapper change.
• app / prompt: a single product input/output behavior must remain stable.

Use cautilus discover scenarios only when you need the proposal-input normalization catalog in YAML, or cautilus discover scenarios --format json when a parser needs JSON. Use cautilus doctor --repo-root . when the selected branch is repo evaluation readiness. Use cautilus doctor --repo-root . --scope agent-surface or doctor --scope agent-surface when the selected branch is only local Cautilus Agent install/readiness.

Runner Readiness

Runner readiness is a setup substrate under claim, eval, and improve, not a fourth job, so orient it from doctor status runnerReadiness and its next branches and treat missing, smoke-only, stale, or not-ready states as the blocker to clear before app behavior proof. When app proof is required, help the user build a headless product runner that reuses the real product path instead of a prompt-only mock, and help produce a cautilus.runner_assessment.v1 at the binary-named scaffold path, letting the binary own the scaffold source and fields rather than authoring from prose. Keep the proof class honest and stop improve when the selected claim needs runner-backed proof that is absent, stale, or only fixture-smoke or coding-agent-messaging; only in-process-product-runner or live-product-runner with a ready-for-selected-surface assessment may back an app behavior-change claim. See runner-readiness.md for the build and assessment sequence and the assessment fields.

Workflow

1. Start from doctor status, an explicit user branch, or an existing Cautilus packet.
2. Restate the selected claim, baseline, intended behavior, and decision boundary.
3. Use workspace start for a multi-command run instead of inventing unrelated /tmp paths by hand.
4. Use evaluate comparison prepare when the run needs clean git-ref A/B workspaces.
5. Run adapter-defined preflight commands before long evaluations.
6. Run cautilus evaluate fixture for checked-in fixtures and read eval-summary.json as the first bounded evaluation decision.
7. Build report.json only when the workflow needs the broader report/review/evidence/improve packet layer.
8. If the adapter defines executor_variants, run cautilus evaluate review variants instead of retyping ad hoc shell commands. If evaluate review variants are requested but unavailable on the selected adapter, treat that as a gate defect to fix or explicitly waive before release.
9. Use discover scenarios propose when normalized proposal candidates already exist and the next move is a checked-in scenario packet.
10. Use improve or GEPA-style search only after the claim and held-out proof surface are explicit.
11. Report exact commands, exact adapter selection, exact artifact paths, and the final recommendation.

When the target repo is Cautilus itself, prefer the checked-in self-dogfood wrappers over rebuilding the mode/report/review chain by hand; see self-dogfood-runner.md for wrapper entries and claim boundaries.

Packet Reading

Use product-owned outputs instead of paraphrasing command results from memory.

• cautilus.agent_status.v1: no-input orientation over binary health, agent surface, adapter, claim state, scan scope, and next branches.
• cautilus.claim_proof_plan.v1: source-ref-backed claim candidates and proof-layer routing, not a verdict.
• cautilus.claim_status_summary.v1: grouped status from an existing claim packet.
• cautilus.claim_validation_report.v1: claim packet and evidence-ref validation, not evidence discovery.
• cautilus.claim_eval_plan.v1: intermediate eval-fixture planning over reviewed claims, not a host fixture writer.
• eval-cases.json: product-normalized test cases handed to the host-owned runner.
• eval-observed.json: observed behavior packet written by the runner.
• eval-summary.json: first bounded evaluation decision packet.
• report.json, review.json, review-summary.json, evidence-bundle.json: broader decision packets for review, evidence, and improvement workflows.
• improve-input.json, improve-search-result.json, improve-proposal.json, revision-artifact.json: bounded improvement and handoff packets.

Humans usually read HTML renders first, then the underlying packet if they need exact fields. Agents should read the packet first, then cite HTML only when a browser view is the deliverable.

Guardrails

• Keep host-repo fixtures, prompts, wrappers, and acceptance policy outside the product boundary.
• Prefer CLI examples and schemas over hand-written packet JSON.
• Treat deterministic tests and CI checks as host-owned proof when they are cheaper and stronger than an LLM behavior evaluation.
• Treat review loops and improver output as bounded decision artifacts, not open-ended retries.