boundary-audit

name	boundary-audit
description	Use when running the quarterly boundary audit of this repo's external assumptions — inventory the write/read/docs surfaces against the conformance ledger, track the verification-class trend, retro-check recent bugs for missing class-level ratchets, and spot-check docs freshness. Produces a dated report under docs/audits/ plus filed follow-up issues. Repo-maintenance skill, not a finance skill.

Boundary Audit

Quarterly assumption audit (Epic D, issue #445). Gates catch anticipated drift; this ritual catches the unanticipated. It reads the conformance ledger (src/conformance/ledger.ts), the codebase, and recent GitHub history — never the user's financial data.

When to use

• A quarter (~90 days) has passed since the newest report in docs/audits/
• After a boundary incident (a bug caused by Copilot's server or cache drifting from our local model), to retro-check the whole boundary, not just the incident
• The user says "boundary audit" or asks how trustworthy our external assumptions currently are

Do NOT use if

• The user wants to verify one specific enum/field against production → run the relevant scripts/smoke/ script (bun run smoke) instead
• The user wants to audit their transactions or finances → /finance-cleanup or /finance-pulse; this skill never touches financial data

Guardrails

1. Read-only toward Copilot. The audit reads source, the ledger, and GitHub. Never call MCP write tools, never run live probes against app.copilot.money — if an assumption needs a live probe, that is a filed follow-up issue, not an audit step.
2. The report is committed — PII rules apply. No real financial figures, account balances, account names, or transaction details anywhere in the report or in filed issues. The audit's subject matter is schemas and processes, so this should be natural; if an example needs a figure, use a placeholder ("$X", synthetic numbers).
3. Findings become issues, not inline fixes. The audit PR contains the report (and nothing else). Every actionable finding is filed as a GitHub issue; fixing them happens in their own PRs. Exception: none — even one-line doc staleness gets an issue, so the report's findings list and the issue tracker stay 1:1.
4. Dedupe before filing. Search open issues (gh issue list --search) before creating one; if an existing issue covers the finding, reference it in the report instead of filing a duplicate.
5. No invented history. Phase 3 reviews only PRs/issues that actually exist; quote numbers, not memories.

Phase 0 — Baseline

1. Run from the repo root; bun install if node_modules is missing.

2. Find the previous report: newest docs/audits/*-boundary-audit.md. Note its date (LAST_AUDIT) and its recorded class distribution. If no report exists, this is the first run: set LAST_AUDIT to 90 days ago and treat every "vs last audit" comparison as "baseline established".

3. Confirm the ledger gate is green (it also runs in CI):

   bun test tests/conformance/
   

Phase 1 — Inventory: assumptions vs ledger

Enumerate every class of external assumption and diff against the ledger. Anything external, new, and unlisted is a finding.

1. Dump the ledger's surfaces for comparison:

   bun -e "const m = await import('./src/conformance/ledger.ts'); for (const e of m.CONFORMANCE_LEDGER) console.log(e.class.padEnd(13), e.kind.padEnd(14), e.surface);"
   

2. Enumerate every GraphQL operation the code can send. All operation documents live in src/core/graphql/operations.generated.ts (generated by bun run generate:graphql from docs/graphql-capture/operations/):

   grep -hoE "(query|mutation) [A-Za-z0-9_]+" src/core/graphql/operations.generated.ts | sort -u
   

   Note the operation name (e.g. EditBudget) can differ from the mutation field name the ledger uses (Mutation.editCategoryBudget) — resolve via the wrapper module in src/core/graphql/ before declaring a mismatch.

3. Write surface (mutations). Bidirectional ledger coverage is enforced by tests/conformance/ledger.test.ts (Phase 0 confirmed it green), so this is automated — but check the escape hatch: GraphQL wrapper modules whose mutations have no MCP tool (no toolParams, e.g. editAccount) are NOT caught by the test. Confirm each mutation from step 2 appears in the ledger dump.

4. Read surface (queries). The ledger test does not reach these. Diff the query operations from step 2 against the ledger. Each query operation, its response shape (the hand-written TS interfaces in src/core/graphql/queries/*.ts), and any enum/string-union it sends (see src/core/graphql/queries/_shared.ts) is an external assumption. Unlisted → finding.

5. Cache decoder surface. src/core/decoder.ts assumes Copilot's local Firestore/LevelDB doc shapes — a different external boundary with its own controls (warnUnreadFields, bun run scripts/decode-coverage.ts). Note in the report whether those controls still run and whether any new collection processor was added since LAST_AUDIT without coverage. Unledgered cache assumptions are a finding only if no control covers them.

6. Docs facts. External-facing factual claims in CLAUDE.md / README.md / CONTRIBUTING.md (endpoints, paths, counts, flag behavior) are assumptions too; the freshness check is Phase 4 — here, only note any claim about Copilot's API that the ledger should own but doesn't.

Phase 2 — Class distribution trend

The ledger is the source; the trend is the health metric.

bun -e "const m = await import('./src/conformance/ledger.ts'); console.log(m.formatClassDistribution());"

Record the table in the report next to LAST_AUDIT's numbers. Findings:

• unverified count grew with no corresponding open follow-up issue
• gated share shrank (an oracle was removed or downgraded)
• Any unverified entry that has survived two consecutive audits with no filed issue (i.e. it appeared unverified in the previous report and remains unverified now) → file one (per-entry, with the surface name in the title)

Phase 3 — Bug-class review (retroactive ratchet check)

Apply the Bug Response Ritual questions (CONTRIBUTING.md, from D1 #444) to every incident since LAST_AUDIT — read them together, not one-by-one; bug-class patterns only show up in the aggregate.

1. Collect incidents:

   gh pr list --search "merged:>=LAST_AUDIT" --limit 200 --json number,title,mergedAt
   gh issue list --state all --label bug --search "created:>=LAST_AUDIT" --json number,title,state
   

   Keep PRs whose titles/bodies are fix:-shaped or reference a bug issue. Don't trust the bug label alone — bug reports sometimes land unlabeled, so also scan the merged-PR list for incident-shaped titles (fix, regression, "wrong", "missing") regardless of label.

2. For each incident, answer in the report (one table row each):

   • Class — name the bug class, not the instance (e.g. "local enum transcribed wrong", "name→ID resolution skipped", "nullability stricter than reality")
   • Detector? — does a class-level detector now exist (smoke gate, ledger oracle, schema test, lint rule)? Cite it, or mark "instance-only regression test" / "none"
   • Siblings? — was the same class checked across sibling surfaces, or only where it bit?

3. Missing class-level detector or unchecked siblings → finding → issue, linking the "Bug Response Ritual" template in CONTRIBUTING.md (the audit questions above are that template, applied retroactively).

Phase 4 — Docs freshness

Beyond what tests/unit/doc-sync.test.ts automates (tool-count phrases, removed-path references), spot-check the rest of CLAUDE.md against the code:

• Every file/directory path mentioned exists (ls them)
• Every bun run <script> mentioned exists in package.json scripts, and prose descriptions of compound scripts (e.g. what check runs) match the actual script definition
• Headline behavior claims (which flags imply which, which tools swap in live mode, database location shape) match src/server.ts / src/cli.ts
• README.md and CONTRIBUTING.md: same treatment for their headline sections

Each stale fact is a finding. If a whole category of fact keeps rotting, also consider a finding for "automate this in doc-sync" (that's the ratchet).

Phase 5 — Report

Write docs/audits/<YYYY-MM-DD>-boundary-audit.md (today's date) using this structure:

# Boundary audit — <YYYY-MM-DD>

Previous audit: <date or "none (first run)">. Window reviewed: <range>.

## 1. Inventory diff (assumptions vs ledger)
<per-surface-class result: covered / uncovered, with counts>

## 2. Class distribution
| class | now | last audit | delta |
<table + one-line trend verdict>

## 3. Bug-class review
| incident | class | class-level detector | siblings checked |
<table + aggregate pattern observations>

## 4. Docs freshness
<stale facts found, or "clean">

## 5. Findings & filed issues
| # | finding | severity | issue |
<every finding, each with an issue number or an explicit "not filed because…">

PII check before commit: grep the report for \$[0-9], 4-digit-plus integers in financial context, and account/bank names.

Phase 6 — File issues and open the PR

1. File one issue per actionable finding (after the Guardrail-4 dedupe search). Title prefix boundary-audit:; label audit plus area:boundary when it concerns the ledger/GraphQL surface, documentation for docs rot; body links the report path and the audit date.
2. Backfill the issue numbers into the report's findings table.
3. Branch, commit the report (only the report; commit-message conventions — trailers, PII rules — per CLAUDE.md), open a ready-for-review PR titled audit: boundary audit <YYYY-MM-DD>, and reference the filed issues in the body (plain references — do not write "Closes #N"; the issues stay open until their own PRs land).