| name | crouton-harness-observability |
| layer | stack |
| description | How to MEASURE this repo instead of eyeballing it — interpret the loop-station context-budget trend (history.jsonl, thresholds, the tokenizer-switch gotcha), read a session trace, use the eval ledger (cost-per-success), and pick the right evidence tool (app-shots, smoke-deployed, db-counts, --check modes). Use when asked "is the harness bloating", "how big is CLAUDE.md really", "did this agent run actually succeed", "prove this works", "what evidence do we have", or when tempted to declare a capability (browser, tool) unavailable. For RUNNING the loop-station scripts, defer to the loop-station skill — this skill is interpretation + the wider evidence toolkit. |
crouton-harness-observability
One line: the repo's proof toolkit — where the measurements live, how to read them without fooling yourself, and which gaps still force eyeballing.
"The harness" = this repo's agent-operating layer: root CLAUDE.md + AGENTS.md + .claude/skills/** + .claude/agents/** + the LLM CI workflows. It is treated as a system worth measuring (AGENTS.md "Observe the harness").
When to use / when NOT to use
| You want to… | Use |
|---|
| Interpret a loop-station record, the budget trend, a session trace, the eval ledger; pick an evidence tool; know what is NOT instrumented | this skill |
| Run the loop-station scripts (gather/append/parse/advisor mechanics, CI wiring) | loop-station skill (it owns the runbooks; this skill never restates them) |
| Boot/observe an app locally or on staging, screenshots, DB inspection runbook | crouton-run-and-operate |
| Run tests + the honest coverage picture | crouton-validation-reality |
| Understand which workflow produced a CI artifact | crouton-ci-and-deploy-map |
| Know which config file a measurement reads | crouton-config-registry |
1. The loop-station inventory (WS1) — reading it
Data: writeups/loop-station/history.jsonl — committed on purpose (tiny, git blame-able; see its README). One JSON record per merge that touches the harness, appended by CI (.github/workflows/loop-station-inventory.yml). The WS2 runtime trace is the only loop-station dataset that is gitignored.
Record shape and thresholds are owned by the loop-station skill (.claude/skills/loop-station/SKILL.md; threshold bands in .claude/skills/loop-station/lib/scorecard.mjs THRESHOLDS — tune only there). Don't restate them here. The one field to check before reading any number is tokenizer — anthropic (API count_tokens) vs heuristic (offline fallback). That field is the ruler.
Current numbers: run node .claude/skills/loop-station/gather.mjs --pretty — never quote a remembered measurement (no API key in the env → heuristic tokenizer, so compare only against heuristic-era records).
The tokenizer discontinuity — the #1 misread
Every record carries tokenizer. Never compare numbers across two tokenizers. The heuristic overcounts vs Anthropic count_tokens by roughly 20–25% here. Two real traps in the committed history:
- The apparent budget "drop" at the heuristic→anthropic switch is the ruler changing, not a diet. (The API tokenizer landed in PR #958; the switch shows up at the first record where CI actually ran with
ANTHROPIC_API_KEY — a later PR — so the step and the code change don't share a commit.)
- A record can silently revert to
heuristic when CI runs without the key (it happened; the point reads red). That is missing-secret noise, not a regression — check the surrounding anthropic-era points before ticketing.
The advisor (advisor.mjs, see loop-station skill) encodes this rule: deltas are compared only within the same tokenizer. A large corpus step can also be deliberate (e.g. a batch of new skills landing) — read the PR behind the record before calling it drift.
2. The session trace (WS2) — reading it at a glance
Producer mechanics live in the loop-station skill (parse-transcripts.mjs, collect-traces.mjs). What you get is trace.jsonl of events:
{ ts, kind, name, parent, depth, agentId?, durMs? }
kind = skill / agent / tool; parent + depth reconstruct the real call tree (2+-level agent recursion supported).- Privacy rule: names + correlation ids + durations only — never tool inputs or results. Runtime exhaust → gitignored; CI ships it as an artifact tagged by
run_id (the two-step collect→upload snippet lives in claude.yml).
- How to read one fast: sort by
ts, indent by depth. Look for (a) the same skill/agent firing repeatedly at the same depth = a retry loop; (b) a long durMs on an agent with few children = it stalled, not worked; (c) tools firing with no skill parent = the model bypassed the skill it should have loaded.
- pi-harness runs (
pi-telemetry.mjs, #944) emit the same event shape but at agent granularity only — no tool-level nesting until pi-otel spans are wired (open).
3. The eval ledger — did the run actually succeed, and at what cost?
Data: writeups/reports/eval-ledger.jsonl (committed, append-only). Scripts: scripts/eval-ledger/{schema,append,scoreboard}.mjs — usage in scripts/eval-ledger/README.md:
node scripts/eval-ledger/scoreboard.mjs
node scripts/eval-ledger/append.mjs --flow a11y-reports --harness pi \
--model anthropic/claude-haiku-4-5-20251001 --outcome report
Semantics that matter when reading it:
- One record per agent run:
{ ts, flow, skill, harness, model, cost_usd, turns, wall_s, artifact_gate, ci, outcome, human, fix_rounds, ref, notes }.
- Success = no gate failed AND
outcome ∈ {merged, report}. The routing metric is cost-per-success, never raw cost or vibes. model is provider-qualified (anthropic/… vs ollama/…) because it's a cost board.
- The gold negative: the seed record is the 2026-06-22 pi/Haiku decompose spike that merged unreviewed to main with full creds and was reverted in PR #862 (
outcome: "reverted", human: "down"). outcome: reverted is the strongest failure signal the ledger has; it birthed the reports-only-first rule (#867). Never delete or "clean up" that row.
- The ledger feeds
.claude/routing.json (model routing, #864/#865) — routing changes should cite ledger evidence, the way the orchestrator/decomposer opus→sonnet demotion cited an N=4 A/B.
- Weekly scoreboard post:
.github/workflows/eval-scoreboard.yml, cadence config-as-data in .github/digests.yml (see crouton-config-registry).
4. Evidence tools — pick by what you must prove
Full operating runbooks for the app-facing ones are in crouton-run-and-operate; this table is the "which proof do I reach for" index.
| To prove… | Tool (exact invocation) | Notes |
|---|
| "the UI renders like this" | node scripts/app-shots.mjs <baseUrl> <path[:name]> […] [--out <dir>] | Uses preinstalled chromium at /opt/pw-browsers (globs newest build — never hardcode a build number); output screenshots/<name>.png (HARD GATE location); exit 1 on any failure |
| "the DEPLOY actually works" (not just built) | node scripts/smoke-deployed.mjs --url <url> --email <e> --password <pw> [--app <n>] [--manifest <app>/deploy.config.json] | Login proof via /api/auth/get-session → optional CRUD round-trip from deploy.config.json smoke.crud → screenshot. Built because "typecheck + boot" twice masqueraded as done (#293, per its header). Report-only in CI unless smoke.required: true |
| "a generated app still boots/auths/CRUDs" | e2e fixture harness — see crouton-validation-reality and the e2e-smoke skill | Trace/video/screenshots land in playwright-report/; CI artifact visual-qa-<fixture> (14-day) |
| "types still hold" | pnpm typecheck (never npx nuxt typecheck from root) | Necessary, never sufficient — AGENTS.md "Done is signed off, not asserted"; the story behind it: crouton-failure-archaeology (#988) |
| "what data is on a remote DB" (read-only) | node scripts/db-counts.mjs --app <app> --env staging|prod [--dry-run] | SELECTs only; discovers db name from wrangler.jsonc; real run needs CF creds, --dry-run needs none |
| "which gates apply to this path" | node scripts/harness-stages.mjs <path> | e.g. packages/crouton-core/x.ts → stage: package, gates(required): test-first |
| "layer tags are honest" | node scripts/harness-layers.mjs --check | ⚠️ Despite its own header it is wired into no CI workflow, and it can be red for pre-existing untagged skills — a failure here may not be yours (re-verify block; #1098) |
| "generated docs aren't stale" | node scripts/gen-skills-doc.mjs --check · node scripts/gen-routing.mjs --check · node scripts/gen-package-catalog.mjs --check | skills-doc: a skill missing from META only warns; stale HTML fails — new skills trip it until registered + regenerated |
| "is a budget regression worth a ticket" | node .claude/skills/loop-station/advisor.mjs --pretty | Deterministic gate over history.jsonl; cross-check any always-on red against the tokenizer rule (§1) before ticketing |
5. Verify capabilities, don't assume (#629)
Root CLAUDE.md ("You HAVE a headless browser") is the standing rule; the incident behind it is #629: an agent checked the default Playwright path (empty), saw npx playwright install fail (egress-blocked CDN), and declared "no browser" — while chromium sat preinstalled at /opt/pw-browsers. Two partial negatives are not proof of absence. Before declaring anything unavailable: env vars → non-default paths → filesystem search → system binaries. Same doctrine applies to TodoWrite (absent on some harnesses — fall back per root CLAUDE.md, don't retry-spam) and any "X isn't available here" claim inherited from a prior session: probe for 5 seconds first. app-shots.mjs's findChromium() is the codified fix (env override: PLAYWRIGHT_CHROMIUM_PATH).
6. What has NO instrumentation (open gaps)
| Gap | Status |
|---|
Staging Worker logs: only a couple of apps have a logs script, and those tail the production worker (npx wrangler tail <app>). No staging tail script exists; hand-type npx wrangler tail <app>-staging --env staging. No log persistence (no Logpush/Sentry) — an untailed Worker error is gone forever | open |
Runtime product analytics: packages/crouton-analytics exists (PostHog default, epic #945/#946) but no app/poc/fixture depends on it (re-verify block). No uptime checks; deployed smoke runs only on deploys | open |
Visual regression: screenshots exist, comparison is human — pixel-diff baselines explicitly deferred (e2e/CLAUDE.md) | open, deliberate |
harness-layers --check in CI: documented as CI-enforced, actually unwired (§4; #1098) | open drift |
| pi WS2 tool-level nesting: agent-granularity only until pi-otel spans are collected | open |
CI-agent tokenizer flakiness: loop-station CI records silently fall back to heuristic when the key is missing — a red in history may be a missing secret, not growth | known noise mode |
Provenance and maintenance
verified: 2026-07-02
Facts checked against: writeups/loop-station/history.jsonl (all records read) + its README; live gather.mjs --pretty and advisor.mjs --pretty runs; .claude/skills/loop-station/SKILL.md + lib/scorecard.mjs + parse-transcripts.mjs; scripts/eval-ledger/{README.md,append.mjs,scoreboard.mjs} + writeups/reports/eval-ledger.jsonl; headers of scripts/{app-shots,smoke-deployed,db-counts}.mjs; live runs of the --check modes; GitHub issue #629; greps for crouton-analytics consumers and "logs" scripts. Unverified-but-cited: issue numbers inside quoted script/skill prose (#293, #867, #944 etc.) — taken from the cited files themselves. Volatile facts (token counts, record counts, exit codes, issue states) — re-verify here, not in the prose:
node .claude/skills/loop-station/gather.mjs --pretty
grep -o '"tokenizer":"[a-z]*"' writeups/loop-station/history.jsonl | tail -3
sed -n '/THRESHOLDS/,/^}/p' .claude/skills/loop-station/lib/scorecard.mjs
node scripts/eval-ledger/scoreboard.mjs
node scripts/harness-layers.mjs --check; grep -rl harness-layers .github/ || echo "still unwired"
grep -rl crouton-analytics --include=package.json apps pocs fixtures || echo "still unconsumed"
gh issue view 629 --json state -q .state
grep -l '"logs"' apps/*/package.json
