crouton-harness-observability

name	crouton-harness-observability
layer	stack
description	How to MEASURE this repo instead of eyeballing it — interpret the loop-station context-budget trend (history.jsonl, thresholds, the tokenizer-switch gotcha), read a session trace, use the eval ledger (cost-per-success), and pick the right evidence tool (app-shots, smoke-deployed, db-counts, --check modes). Use when asked "is the harness bloating", "how big is CLAUDE.md really", "did this agent run actually succeed", "prove this works", "what evidence do we have", or when tempted to declare a capability (browser, tool) unavailable. For RUNNING the loop-station scripts, defer to the loop-station skill — this skill is interpretation + the wider evidence toolkit.

crouton-harness-observability

One line: the repo's proof toolkit — where the measurements live, how to read them without fooling yourself, and which gaps still force eyeballing.

"The harness" = this repo's agent-operating layer: root CLAUDE.md + AGENTS.md + .claude/skills/** + .claude/agents/** + the LLM CI workflows. It is treated as a system worth measuring (AGENTS.md "Observe the harness").

When to use / when NOT to use

You want to…	Use
Interpret a loop-station record, the budget trend, a session trace, the eval ledger; pick an evidence tool; know what is NOT instrumented	this skill
Run the loop-station scripts (gather/append/parse/advisor mechanics, CI wiring)	loop-station skill (it owns the runbooks; this skill never restates them)
Boot/observe an app locally or on staging, screenshots, DB inspection runbook	crouton-run-and-operate
Run tests + the honest coverage picture	crouton-validation-reality
Understand which workflow produced a CI artifact	crouton-ci-and-deploy-map
Know which config file a measurement reads	crouton-config-registry

1. The loop-station inventory (WS1) — reading it

Data: writeups/loop-station/history.jsonl — committed on purpose (tiny, git blame-able; see its README). One JSON record per merge that touches the harness, appended by CI (.github/workflows/loop-station-inventory.yml). The WS2 runtime trace is the only loop-station dataset that is gitignored.

Record shape and thresholds are owned by the loop-station skill (.claude/skills/loop-station/SKILL.md; threshold bands in .claude/skills/loop-station/lib/scorecard.mjs THRESHOLDS — tune only there). Don't restate them here. The one field to check before reading any number is tokenizer — anthropic (API count_tokens) vs heuristic (offline fallback). That field is the ruler.

Current numbers: run node .claude/skills/loop-station/gather.mjs --pretty — never quote a remembered measurement (no API key in the env → heuristic tokenizer, so compare only against heuristic-era records).

The tokenizer discontinuity — the #1 misread

Every record carries tokenizer. Never compare numbers across two tokenizers. The heuristic overcounts vs Anthropic count_tokens by roughly 20–25% here. Two real traps in the committed history:

• The apparent budget "drop" at the heuristic→anthropic switch is the ruler changing, not a diet. (The API tokenizer landed in PR #958; the switch shows up at the first record where CI actually ran with ANTHROPIC_API_KEY — a later PR — so the step and the code change don't share a commit.)
• A record can silently revert to heuristic when CI runs without the key (it happened; the point reads red). That is missing-secret noise, not a regression — check the surrounding anthropic-era points before ticketing.

The advisor (advisor.mjs, see loop-station skill) encodes this rule: deltas are compared only within the same tokenizer. A large corpus step can also be deliberate (e.g. a batch of new skills landing) — read the PR behind the record before calling it drift.

2. The session trace (WS2) — reading it at a glance

Producer mechanics live in the loop-station skill (parse-transcripts.mjs, collect-traces.mjs). What you get is trace.jsonl of events:

{ ts, kind, name, parent, depth, agentId?, durMs? }

• kind = skill / agent / tool; parent + depth reconstruct the real call tree (2+-level agent recursion supported).
• Privacy rule: names + correlation ids + durations only — never tool inputs or results. Runtime exhaust → gitignored; CI ships it as an artifact tagged by run_id (the two-step collect→upload snippet lives in claude.yml).
• How to read one fast: sort by ts, indent by depth. Look for (a) the same skill/agent firing repeatedly at the same depth = a retry loop; (b) a long durMs on an agent with few children = it stalled, not worked; (c) tools firing with no skill parent = the model bypassed the skill it should have loaded.
• pi-harness runs (pi-telemetry.mjs, #944) emit the same event shape but at agent granularity only — no tool-level nesting until pi-otel spans are wired (open).

3. The eval ledger — did the run actually succeed, and at what cost?

Data: writeups/reports/eval-ledger.jsonl (committed, append-only). Scripts: scripts/eval-ledger/{schema,append,scoreboard}.mjs — usage in scripts/eval-ledger/README.md:

node scripts/eval-ledger/scoreboard.mjs          # markdown rollup; --json / --html
node scripts/eval-ledger/append.mjs --flow a11y-reports --harness pi \
  --model anthropic/claude-haiku-4-5-20251001 --outcome report   # append one run; --check validates only

Semantics that matter when reading it:

• One record per agent run: { ts, flow, skill, harness, model, cost_usd, turns, wall_s, artifact_gate, ci, outcome, human, fix_rounds, ref, notes }.
• Success = no gate failed AND outcome ∈ {merged, report}. The routing metric is cost-per-success, never raw cost or vibes. model is provider-qualified (anthropic/… vs ollama/…) because it's a cost board.
• The gold negative: the seed record is the 2026-06-22 pi/Haiku decompose spike that merged unreviewed to main with full creds and was reverted in PR #862 (outcome: "reverted", human: "down"). outcome: reverted is the strongest failure signal the ledger has; it birthed the reports-only-first rule (#867). Never delete or "clean up" that row.
• The ledger feeds .claude/routing.json (model routing, #864/#865) — routing changes should cite ledger evidence, the way the orchestrator/decomposer opus→sonnet demotion cited an N=4 A/B.
• Weekly scoreboard post: .github/workflows/eval-scoreboard.yml, cadence config-as-data in .github/digests.yml (see crouton-config-registry).

4. Evidence tools — pick by what you must prove

Full operating runbooks for the app-facing ones are in crouton-run-and-operate; this table is the "which proof do I reach for" index.

To prove…	Tool (exact invocation)	Notes
"the UI renders like this"	node scripts/app-shots.mjs <baseUrl> <path[:name]> […] [--out <dir>]	Uses preinstalled chromium at /opt/pw-browsers (globs newest build — never hardcode a build number); output screenshots/<name>.png (HARD GATE location); exit 1 on any failure
"the DEPLOY actually works" (not just built)	node scripts/smoke-deployed.mjs --url <url> --email <e> --password <pw> [--app <n>] [--manifest <app>/deploy.config.json]	Login proof via /api/auth/get-session → optional CRUD round-trip from deploy.config.json smoke.crud → screenshot. Built because "typecheck + boot" twice masqueraded as done (#293, per its header). Report-only in CI unless smoke.required: true
"a generated app still boots/auths/CRUDs"	e2e fixture harness — see crouton-validation-reality and the e2e-smoke skill	Trace/video/screenshots land in playwright-report/; CI artifact visual-qa-<fixture> (14-day)
"types still hold"	pnpm typecheck (never npx nuxt typecheck from root)	Necessary, never sufficient — AGENTS.md "Done is signed off, not asserted"; the story behind it: crouton-failure-archaeology (#988)
"what data is on a remote DB" (read-only)	node scripts/db-counts.mjs --app <app> --env staging|prod [--dry-run]	SELECTs only; discovers db name from wrangler.jsonc; real run needs CF creds, --dry-run needs none
"which gates apply to this path"	node scripts/harness-stages.mjs <path>	e.g. packages/crouton-core/x.ts → stage: package, gates(required): test-first
"layer tags are honest"	node scripts/harness-layers.mjs --check	⚠️ Despite its own header it is wired into no CI workflow, and it can be red for pre-existing untagged skills — a failure here may not be yours (re-verify block; #1098)
"generated docs aren't stale"	node scripts/gen-skills-doc.mjs --check · node scripts/gen-routing.mjs --check · node scripts/gen-package-catalog.mjs --check	skills-doc: a skill missing from META only warns; stale HTML fails — new skills trip it until registered + regenerated
"is a budget regression worth a ticket"	node .claude/skills/loop-station/advisor.mjs --pretty	Deterministic gate over history.jsonl; cross-check any always-on red against the tokenizer rule (§1) before ticketing

5. Verify capabilities, don't assume (#629)

Root CLAUDE.md ("You HAVE a headless browser") is the standing rule; the incident behind it is #629: an agent checked the default Playwright path (empty), saw npx playwright install fail (egress-blocked CDN), and declared "no browser" — while chromium sat preinstalled at /opt/pw-browsers. Two partial negatives are not proof of absence. Before declaring anything unavailable: env vars → non-default paths → filesystem search → system binaries. Same doctrine applies to TodoWrite (absent on some harnesses — fall back per root CLAUDE.md, don't retry-spam) and any "X isn't available here" claim inherited from a prior session: probe for 5 seconds first. app-shots.mjs's findChromium() is the codified fix (env override: PLAYWRIGHT_CHROMIUM_PATH).

6. What has NO instrumentation (open gaps)

Gap	Status
Staging Worker logs: only a couple of apps have a logs script, and those tail the production worker (npx wrangler tail <app>). No staging tail script exists; hand-type npx wrangler tail <app>-staging --env staging. No log persistence (no Logpush/Sentry) — an untailed Worker error is gone forever	open
Runtime product analytics: packages/crouton-analytics exists (PostHog default, epic #945/#946) but no app/poc/fixture depends on it (re-verify block). No uptime checks; deployed smoke runs only on deploys	open
Visual regression: screenshots exist, comparison is human — pixel-diff baselines explicitly deferred (e2e/CLAUDE.md)	open, deliberate
harness-layers --check in CI: documented as CI-enforced, actually unwired (§4; #1098)	open drift
pi WS2 tool-level nesting: agent-granularity only until pi-otel spans are collected	open
CI-agent tokenizer flakiness: loop-station CI records silently fall back to heuristic when the key is missing — a red in history may be a missing secret, not growth	known noise mode

Provenance and maintenance

verified: 2026-07-02

Facts checked against: writeups/loop-station/history.jsonl (all records read) + its README; live gather.mjs --pretty and advisor.mjs --pretty runs; .claude/skills/loop-station/SKILL.md + lib/scorecard.mjs + parse-transcripts.mjs; scripts/eval-ledger/{README.md,append.mjs,scoreboard.mjs} + writeups/reports/eval-ledger.jsonl; headers of scripts/{app-shots,smoke-deployed,db-counts}.mjs; live runs of the --check modes; GitHub issue #629; greps for crouton-analytics consumers and "logs" scripts. Unverified-but-cited: issue numbers inside quoted script/skill prose (#293, #867, #944 etc.) — taken from the cited files themselves. Volatile facts (token counts, record counts, exit codes, issue states) — re-verify here, not in the prose:

node .claude/skills/loop-station/gather.mjs --pretty        # current budget numbers (§1)
grep -o '"tokenizer":"[a-z]*"' writeups/loop-station/history.jsonl | tail -3   # ruler of recent records
sed -n '/THRESHOLDS/,/^}/p' .claude/skills/loop-station/lib/scorecard.mjs      # threshold bands
node scripts/eval-ledger/scoreboard.mjs                     # ledger state + record count
node scripts/harness-layers.mjs --check; grep -rl harness-layers .github/ || echo "still unwired"
grep -rl crouton-analytics --include=package.json apps pocs fixtures || echo "still unconsumed"
gh issue view 629 --json state -q .state                    # capability-assumption incident still open?
grep -l '"logs"' apps/*/package.json                        # which apps have a logs script (2 at verification, both prod-tail)