| name | crouton-validation-reality |
| layer | stack |
| description | How to actually run every kind of test in nuxt-crouton (unit/e2e/typecheck/lint, verified commands) plus the HONEST coverage picture — which packages have real tests vs zero, the dormant skipped/todo cases behind three mocking walls, and what CI truly gates vs what CLAUDE.md claims. Use when asked "how do I run the tests", "is X covered by tests", "what does CI actually check", "why is this suite skipped", "where do I add a test", or before trusting a green build as evidence that something works. |
Validation Reality — what counts as evidence in nuxt-crouton
One-line purpose: the verified test-running commands, the honest coverage map, and the gap between claimed and actual gating — so you know what a green check does and does not prove.
When to use / when NOT to use
| You want | Go to |
|---|
| Run/interpret the fixture smoke end-to-end, pick a fixture, triage its failures | e2e-smoke skill (but see the staleness note in §2 — several of its facts have drifted) |
The test-first sign-off gate — proposing a failing test for human approval before writing packages/* logic | test-review skill |
| Fixture manifest schema, auth-helper internals, adding a fixture | e2e/CLAUDE.md + fixtures/CLAUDE.md (canonical) |
| Which CI workflow a failing check belongs to, deploy pipeline | crouton-ci-and-deploy-map skill |
| A test fails with an unfamiliar error string | crouton-diagnostics-index skill |
| Fresh clone won't build/boot before tests can run | crouton-build-and-env skill |
| This skill | Run any test kind; know what coverage actually exists; know what CI enforces |
1. What counts as evidence here
The house rule: done is signed off, not asserted (AGENTS.md) — green proxies each lied during a real graduation (#988; full story: sibling crouton-failure-archaeology).
Concrete local example: a staging preview deployed green but with an empty database, so a loading-skeleton change "showed no difference" — the deploy proved nothing until the advertised path was walked with real data (#695). So the evidence ladder, weakest → strongest:
pnpm typecheck green — proves types compile, nothing about behaviour.pnpm test green — proves the existing assertions pass (totals: the run summary); see §4 for what has zero assertions.- E2E fixture smoke green — proves a generated app boots → authenticates → does CRUD → mounts package surfaces; it does not deeply drive package UIs (e.g.
with-pages mounts the editor workspace but never types into it).
- Walking the advertised path yourself (the
verify skill's job) and a human sign-off — the only "done".
2. Run every test kind (commands verified — see Provenance)
| Kind | Command | Notes |
|---|
| Unit, all | pnpm test | Vitest. Root vitest.config.ts is a projects config: projects: ['packages/*/vitest.config.ts'] — only packages shipping their own vitest.config.ts participate (count them: ls packages/*/vitest.config.ts). Suite totals (passed/skipped/todo) come from the run summary — re-verify block. |
| Unit, one package | pnpm --filter @fyit/crouton-i18n test | Each participating package has "test": "vitest run". |
| Unit, watch / coverage / UI | pnpm test:watch · pnpm test:coverage · pnpm test:ui | Coverage is v8, per package config. |
| E2E fixture smoke | E2E_FIXTURE=<name> pnpm test:e2e | = playwright test --config e2e/playwright.config.ts; default fixture minimal; one fixture per run (all bind :3000). Full workflow → e2e-smoke skill. |
| E2E, subset of specs | pnpm test:e2e e2e/collection.smoke.spec.ts e2e/surface.smoke.spec.ts | The setup project (login) always runs first via project dependency. |
| Typecheck (the gate) | pnpm typecheck | = pnpm -r --filter './apps/*' typecheck — apps only. NEVER npx nuxt typecheck from root (no app context → thousands of false positives; root CLAUDE.md; mechanics in crouton-build-and-env §4). |
| Typecheck fixtures | pnpm typecheck:fixtures | The #197 generator-regression gate surface. |
| Typecheck MCP | pnpm typecheck:mcp | = pnpm --filter @fyit/crouton-mcp typecheck (was a silent no-op with a stale filter name until fixed under #1098). Stale-name inventory: crouton-config-registry § "Silent no-ops". |
| MCP server tests | pnpm --filter @fyit/crouton-mcp build && pnpm --filter @fyit/crouton-mcp test | Build first — CI does (ci.yml mcp-server-tests job). |
| Lint | pnpm lint / pnpm lint:fix | eslint . from root, eslint.config.mjs. No CI workflow runs repo-wide pnpm lint (the misnamed "Lint & Type Check" job was renamed typecheck-mcp under #1097; CI still runs no eslint). |
| Publish validation | pnpm check:publint · pnpm check:attw | Both target ./packages/crouton only. |
E2E environment realities (verified in e2e/playwright.config.ts):
- Auth env is self-provided:
webServer.env defaults BETTER_AUTH_SECRET=dev, BETTER_AUTH_URL=http://localhost:3000; fixtures also commit a dummy .env. Exported vars override (how CI injects its own). Exporting them is harmless.
- The REAL timeouts are deliberately huge — per-test/navigation/webServer-boot in the minutes,
expect/action in the tens of seconds (read current values with the re-verify grep). The smoke runs against nuxt dev, and cold route compiles on CI take minutes; tightening them is what made the smoke flake (comment in the config itself).
- Sandboxes with blocked browser downloads:
PW_EXECUTABLE_PATH=/opt/pw-browsers/chromium-<build>/chrome-linux/chrome is a committed escape hatch in the config.
- Fresh worktree prerequisite:
pnpm --filter "e2e-fixture-<name>^..." build (dist-consumed @fyit/* deps must exist).
- Shell trap (reproduced):
pnpm test \| tail masks the real exit code — check output content, not $?.
Staleness in the e2e-smoke skill
Its workflow is correct, but several of its facts have drifted — stale timeout claim, fixture table missing the newer fixtures, over-strict "env vars are required" claim — tracked as #1103 (row 7). e2e/CLAUDE.md carries the current story; trust it over the skill's numbers.
3. The fixture harness — what exists and what each proves
Vocabulary: a fixture is a real generated crouton app under fixtures/<name>/ (throwaway, never deployed) that the Playwright harness in e2e/ boots and smokes. What each fixture smokes is declared in its fixtures/<name>/e2e.manifest.json; the specs are generic and manifest-driven — you almost never write per-fixture test code. Full manifest schema + auth internals: e2e/CLAUDE.md (canonical). Delta this skill carries:
Fixtures on disk vs in CI's ALL list — count both fresh (ls fixtures/; grep 'ALL=' .github/workflows/e2e.yml). Snapshot 2026-07-02 — regenerate before relying on it:
| Fixture | Adds | In CI ALL? |
|---|
minimal | core + auth + i18n, one mainItems collection | ✅ |
with-pages | crouton-pages → transitively crouton-editor | ✅ |
with-bookings | crouton-bookings | ✅ |
with-assets | crouton-assets | ✅ |
with-maps | crouton-maps | ✅ |
with-devtools | crouton-devtools + crouton-feedback | ✅ |
with-collab | crouton-collab | ❌ deliberately excluded — pre-existing type errors fail the #197 typecheck gate; #210 keeps it a local-only spike (comment in e2e.yml) |
Manifest keys (typed as FixtureManifest in e2e/helpers.ts): packages (drives smart CI fixture selection, #622), collections[] (key, heading, create, optional update/requiredField/a11y), optional surfaces[], i18n, maps, root a11y. Real example — fixtures/minimal/e2e.manifest.json declares one collection (mainItems, requiredField: "name") plus an i18n flip ("Your teams" → "Jouw teams").
Auth realities in specs (the non-obvious ones; verified in e2e/helpers.ts / e2e/auth.setup.ts):
| Reality | Consequence |
|---|
| Login/register is a RouteModal overlay, not a form page | Helpers fill inputs inside the modal; /auth/login redirects to / and opens it |
| Signup creates no team | ensureTeam() POSTs better-auth org endpoints, which need an Origin header (CSRF) |
| Better-auth 415s bodyless POSTs | Always send Content-Type: application/json (signOut() passes data: {}) |
get-session is cookie-cached (5-min TTL) | Force ?disableCookieCache=true for true state; setup strips the session_data cookie from saved storageState so slow runs don't SSR logged-out |
| Sign-out kills the shared session | Auth specs mint their own browser contexts; content specs call ensureAuthed() (self-healing re-login) |
| Test identity | e2e-user@example.com / TestPassword123!, team e2e-team (constants in e2e/helpers.ts) |
A11y in the smoke (verified in e2e/helpers.ts): axe runs on every collection list + surface; only critical/serious block; color-contrast excluded (theme-owned); baseline of known upstream shell violations = ['aria-allowed-attr'] (#735) — button-name was driven to zero and removed, so regressions there now fail.
CI e2e (.github/workflows/e2e.yml): smart per-PR fixture selection by changed paths (#622); packages in UNIVERSAL (crouton crouton-core crouton-i18n crouton-cli crouton-auth) → full matrix; push to main + nightly cron → always full matrix. Runs in the mcr.microsoft.com/playwright container whose tag must track the repo's @playwright/test version (read both in e2e.yml + the lockfile). Each job regenerates the fixture from committed schemas (crouton-generate.js config --force) and typechecks the regenerated output (the #197 generator-regression gate) before testing — so after changing a generator template, regenerate fixtures and commit, or CI diverges from your tree.
4. The honest coverage table
Test files per package — *.test.ts/*.spec.ts excluding node_modules/dist/.nuxt/.output. Snapshot 2026-07-02 — regenerate with the re-verify block before relying on numbers. At the snapshot, fewer than half the packages had any tests (14 of 31; 17 at zero):
| Package | Test files | | Package | Test files |
|---|
| crouton-core | 29 | | crouton-sales | 2 |
| crouton-layout | 17 | | crouton-mcp | 2 |
| crouton-cli | 15 | | crouton-analytics | 2 |
| crouton-auth | 14 | | crouton-assets | 1 |
| crouton-flow | 8 | | crouton-feedback | 8 |
| crouton-collab | 6 | | crouton-printing | 5 |
| crouton-i18n | 4 | | crouton-triage | 3 |
Zero unit tests at the snapshot: crouton (meta), crouton-admin, crouton-ai, crouton-atelier, crouton-audio, crouton-bookings, crouton-charts, crouton-designer, crouton-devtools, crouton-editor, crouton-email, crouton-events, crouton-maps, crouton-mcp-toolkit, crouton-pages, crouton-themes, crouton-three.
Mitigations and their limits:
pages/bookings/assets/maps/devtools get e2e fixture smoke coverage (§3); editor only transitively (with-pages mounts the workspace). collab has unit tests but no CI e2e (excluded fixture) — and its one component test file is describe.skip'd with a note that it "tests an inline mock component, not the actual CollabEditingBadge.vue" (packages/crouton-collab/tests/components/CollabEditingBadge.test.ts).apps/* and pocs/* have 0 test files (verified by find). Consistent with the stage model (node scripts/harness-stages.mjs apps/velo → test-first opt-in; pocs/* → off) — but it means launched apps (fanfare/velo/triage) have no automated behaviour tests beyond typecheck and fanfare's dual-preset CI build.- The tests that exist are real, not placeholders (spot-checked:
crouton-core/tests/api/*.test.ts, crouton-layout/app/utils/__tests__/layout-viability.test.ts).
Gotcha (reproduced at the snapshot): packages/crouton-charts ships a vitest.config.ts but zero test files — standalone npx vitest run there prints "No test files found, exiting with code 1", while the root projects run tolerates the empty project (root pnpm test is green). Anyone adding --project filtering will hit this.
5. The skipped-suite inventory — three mocking walls
The run summary reports a large skipped + todo tail (exact totals: pnpm test 2>&1 | tail -4; marker counts: the greps in the re-verify block). The dormant cases cluster behind three mocking walls — fixing these mocks is the single highest-leverage test-debt target (on the order of 170 dormant cases at the snapshot, many asserting security-relevant behaviour like team limits):
| Wall | What's dormant | Files (verified describe.skip / todo locations) |
|---|
Better Auth nanostore (+ window.location, WebAuthn) | crouton-auth's entire integration suite (login + registration flows) whole-file skipped; a stack of it.todos in useTeam/useAuth | packages/crouton-auth/tests/integration/auth-registration.test.ts, auth-login.test.ts, tests/unit/composables/useTeam.test.ts, useAuth.test.ts |
Nuxt #imports / useRuntimeConfig | team-utils cases incl. team-limit / allowCreate / single-tenant enforcement; useScopedAccess (import.meta.client unmockable post-import) | packages/crouton-auth/tests/unit/server/team-utils.test.ts, tests/unit/composables/useScopedAccess.test.ts |
| Yjs / WebSocket | crouton-flow's entire realtime-sync feature — unit + integration whole-file skipped, e2e suite skipped, dagre-mock describe.todos → the sync feature has zero passing coverage of any kind | packages/crouton-flow/test/composables/useFlowSync.test.ts, test/integration/flow-sync.test.ts, test/e2e/multiplayer-flow.spec.ts, test/composables/useFlowLayout.test.ts |
Smaller items: crouton-core useCollectionQuery/useCollectionMutation logging todos/skips; crouton-i18n/app/composables/__tests__/zod-sanity.test.ts encodes a do-not-use Zod v4 pattern (z.record(valueSchema) breaks — use the two-arg form) as a skipped sanity test.
6. What CI actually gates vs what CLAUDE.md claims
ci.yml jobs:
| Job | What it actually does | Gap vs the claim |
|---|
typecheck-mcp + typecheck-apps | MCP-server typecheck, and the full app sweep (pnpm typecheck) | Fixed under #1097: typecheck-apps builds the apps' workspace deps, nuxt prepares each app, and runs the exact local sweep — root CLAUDE.md's "EVERY change requires pnpm typecheck" is now a CI gate too (was: lint-and-typecheck, MCP-only, no eslint — renamed since it overclaimed). e2e.yml's regenerated-fixture typecheck (#197) still covers fixtures; the typecheck:mcp script's stale filter was fixed under #1098. |
build-fanfare | Builds fanfare for cloudflare-pages + node-server presets | Fanfare typecheck intentionally NOT gated ("known pre-existing baseline of type errors" — comment in the job); the build is the smoke. |
test | pnpm install --ignore-scripts → builds crouton-auth/crouton-core/crouton → npx nuxt prepare in each apps/* → pnpm test | This is the real unit-test gate. Other suited packages (layout, cli…) run from source; the three builds are for dist-consumers. |
mcp-server-tests | Build then test @fyit/crouton-mcp | — |
changeset-check | Detects packages/ change without a changeset | Emits only ::warning — never fails a PR. |
docs-check / sync-validation / package-check | Docs audit script / field-types sync / publint | — |
Also: repo-wide pnpm lint runs in no workflow (part of the #1097 picture). E2E gating lives in e2e.yml (§3), not ci.yml. For the full workflow map, defer to crouton-ci-and-deploy-map.
Trust order when these conflict: see crouton-docs-trust-map §1. CLAUDE.md's typecheck-everything rule stands as the required working practice; just don't claim CI enforces it.
7. Adding a test the house way
- Check the gate first:
node scripts/harness-stages.mjs <path> → packages/* = test-first required; apps/* opt-in; pocs/* off. For new packages/* logic, the order is owned by the test-review skill: enumerate edge cases in plain language → write the failing test → hold for a lgtm/approve comment (not a reaction/label, #572) → implement to green.
- Placement: match the neighbours —
*.test.ts beside the source or in __tests__/; packages/crouton-core is the reference layout (this is what test-review itself points at).
- If the package has no
vitest.config.ts, add one (copy a sibling, e.g. crouton-i18n) — the root projects glob packages/*/vitest.config.ts picks it up automatically; nothing to register at root.
- Run:
pnpm --filter @fyit/<pkg> test, then root pnpm test, then pnpm typecheck.
- E2E coverage for a package surface: extend the relevant fixture's
e2e.manifest.json (generic specs pick it up) rather than writing a new spec; new fixture → e2e/CLAUDE.md "Adding a fixture" — and remember CI never runs it until its name is appended to ALL in e2e.yml.
Provenance and maintenance
verified: 2026-07-02
pnpm test 2>&1 | tail -4
ls packages/*/vitest.config.ts | wc -l
grep -n 'ALL=' .github/workflows/e2e.yml
grep -nE 'timeout|Timeout' e2e/playwright.config.ts
for p in packages/*/; do echo "$(find $p -path '*node_modules*' -prune -o -type f \( -name '*.test.ts' -o -name '*.spec.ts' \) -print | wc -l) $p"; done | sort -rn | head -20
grep -rn 'describe\.skip' packages/*/test*/ packages/*/tests/ 2>/dev/null | grep -v node_modules
grep -rnE '\b(it|describe|test)\.(todo|skip)\(' packages/*/test*/ packages/*/tests/ 2>/dev/null | grep -v node_modules | wc -l
pnpm typecheck:mcp
