| name | weekly-audit-patterns |
| description | The non-obvious invariants of the proactive weekly Claude audit workflow (.github/workflows/claude-weekly-audit.yml): the stable dedup-key scheme that keeps it from re-filing findings every week, the four audit dimensions (security has its own workflow, claude-security-audit.yml) and which one owns the Fail-Loudly check, and the `bug`-label → auto-fix promotion path. Read before editing that workflow, changing how findings are filed/deduped, or adding an audit dimension. |
Weekly Audit Patterns
.github/workflows/claude-weekly-audit.yml is the repo's one proactive Claude
lens — a scheduled deep review (not triggered by a PR) that fans out one read-only
Claude job per dimension and files a ranked triage issue. Everything else in
claude.yml is reactive. These are the invariants a future editor will otherwise break.
The four dimensions are mutually exclusive
correctness, docs, tests, features — a matrix of one Claude job each. Security
is NOT a dimension here — it moved to its own workflow, claude-security-audit.yml
(deterministic semgrep + a Claude taint/authz/suppression sweep, CVSS-scored, findings to
the private code-scanning tab). Don't re-add it here; that created two half-owners and the
single general "security lens" is what missed the hub tar-slip.
The lenses overlap unless the prompt keeps them disjoint, and the first run proved it:
correctness findings (a rollback that never rolls back, a poller returning null, a mode
that no-ops) leaked into features, and the priciest job's output vanished from the
triage issue. The decisive question for a broken thing: is the code wired but
misbehaving (correctness), never written (features), or contradicted by its docs
(docs)?
correctness owns wired-but-broken behavior AND the CLAUDE.md "Fail Loudly" check
(except Exception: pass, try/except returning a placeholder, silent degradation).features is only genuinely-missing/half-shipped capability — a TODO for code never
written. Wired-but-broken is correctness, not features.docs owns doc-vs-code drift, including a feature documented as working but stubbed.tests in deep mode rolls plain "module X has no coverage" into ONE aggregate finding
(dedup_key: tests:aggregate:untested-modules); separate findings only for risk-bearing
untested logic (auth/gate/precedence/error-mapping/#1655).
Adding a dimension means: add it to the matrix, describe its disjoint lens in the shared
prompt, and the synthesis picks up its findings-<dim>.json automatically.
Published hub agents get the highest bar
Published agents are the shop window — the prompt makes every lens double-check them and
bump any gap up one severity (never 🟡; a default-path break is 🔴). Detect them by a
release_agent_<id>.yml, a shipped SCORECARD.md, or a released version: in
gaia-agent.yaml — currently only the email agent. The bar: in-sync high-quality
README/SPEC.md/SKILL.md/CHANGELOG.md (+ any contract spec) with a real eval SCORECARD.md
(gated by gaia.eval.scorecard_gate, never hand-authored) linked from the README; bulletproof
runtime code (no stubs/silent-fallbacks); solid #1655-grade tests. When a new agent publishes,
the detection generalizes to it automatically — no prompt edit needed.
Severity: 🔴 high · 🟠 medium · 🟡 low — no green
Green (🟢) reads as "pass/good," so it's banned. Broken behavior always outranks a
missing test — never rate "module X has no tests" above a feature that's actually
broken. High = security / data loss / default-path break; medium = broken user-facing
behavior, a false doc, or a missing test guarding auth/a gate/destructive logic; low =
missing tests on non-risk logic, cosmetic gaps. The synthesis emits a section per
dimension (fixed order: Correctness, Features, Docs, Tests), grouping each
finding under the dimension it declares — it never re-buckets.
Child issues are 🔴/🟠 only, and tagged auto-fixable
Only high/medium findings get a child issue (and thus one-click bug→auto-fix promotion).
🟡 (low) findings are listed in the parent triage issue and nowhere else — this caps
tracker churn (the first deep run filed 19 children, ~13 of them low-value coverage nits).
Each finding carries an auto_fixable boolean; the child body says whether applying bug
will let auto-fix land it (locatable/small) or whether it needs a human (a test suite, a
refactor) — so maintainers don't promote something auto-fix can't handle.
Precision gate — a false finding erodes the whole audit
Recall is cheap; trust is not. Each finding must carry evidence (a concrete quote or
path:line the dimension actually read), and the dimension prompt requires verifying the
problem is present AND not already handled elsewhere before reporting. The synthesis
drops any 🔴/🟠 finding whose evidence doesn't substantiate its title, and does an
intra-run cross-dimension dedup (a stubbed command flagged by both docs and
correctness is ONE issue, not two — keep the most severe, note the other lens).
Dedup + suppression — the single biggest usability risk
Each finding carries dedup_key = <dimension>:<repo-relative-path>:<symbol-or-section>.
The symbol is a function/class name or doc heading — NEVER a line number (line numbers
move, so a line-based key re-files the same finding every week and the issue is unusable by
week 3). The key is embedded in each child body as <!-- audit-key: KEY -->. Synthesis
skips a finding whose key is in EITHER set:
- already-filed — keys on any open
weekly-audit issue (avoid duplicates).
- suppressed-forever — keys on any
weekly-audit issue also labeled audit-wontfix
(open or closed). This is how you permanently silence accepted debt: close a child with
audit-wontfix and it never comes back. Without this, wontfix findings resurface every
deep run forever.
Parent triage issues accumulate — the workflow NEVER closes one
Each run files a NEW parent (Weekly audit — <mode> — <run_id>) and cross-links the
most recently created open parent with a comment ("Follow-up audit run filed as #N —
stays OPEN until its findings are addressed") — it does not close it. An earlier
version auto-closed the previous parent as "superseded," which silently hid an epic's
still-open child findings the moment the next run fired (#2010: 18 unaddressed children
went dark). Only a human maintainer may close a parent, and only after its findings
are addressed. Child issues stay open regardless (they're the actionable units) — that
was already true and is unchanged.
Since parents are never auto-closed, there can be several open Weekly audit — issues at
once; synthesis always cross-links the highest-numbered (most recent) one, never assumes
there's "at most one." The tradeoff is deliberate: parents pile up until a maintainer
closes them, trading tracker tidiness for guaranteeing unaddressed findings stay visible.
The parent opens with a one-line tally (new/low/suppressed counts) for trend. On a
zero-new-findings run, synthesis posts nothing and does not touch any prior parent.
Security is out of scope here — it has its own workflow
Security moved to .github/workflows/claude-security-audit.yml (see its own patterns
skill). This audit files to a public triage issue, which is the wrong channel for a
vulnerability — so the prompt now tells every lens to hand off any security issue it
notices rather than file it, and the synthesis emits no security section. Do not
re-add a security dimension here or route a security finding into the public issue.
Promotion: bug label → existing auto-fix job
Child issues are opened without any auto-fix trigger label. A maintainer promotes
one to a PR by applying the bug label; the existing auto-fix job in
claude.yml (gated on label.name == 'bug' and contains(labels,'bug')) then
creates the branch + PR. There is no PR-creation code in this workflow — humans
gate every code change. A documentation/tests label alone does NOT trigger auto-fix;
route promotions through bug unless you deliberately widen the auto-fix if.
Cost & safety invariants
- Model is
AUDIT_MODEL (top-level env, claude-opus-4-8 — ~half the token burn of
Fable for comparable static-review quality). One place to change it; swap to
claude-fable-5 for maximum depth at ~2x cost. A measured Fable deep run was ~$45 of
API-equivalent subscription usage; Opus roughly halves that.
- Serialized dimensions: the matrix runs
max-parallel: 1 so the run is a steady
drip, not a 4-job burst — this keeps it under the Max subscription's rolling (5-hour)
rate limit. If you re-parallelize, expect a token spike that can trip that limit.
- Skip-if-empty: normal mode exits in
preflight before any Claude call on a
no-change week. Deep mode never skips.
- Modes:
normal = last N days' diff; deep = whole codebase, auto-selected on the
first Monday of the month (day-of-month ≤ 7, since cron only fires Mondays).
- Read-only: dimension + synthesis jobs run
--allowedTools Read,Grep,Glob,Bash —
no Edit/Write, never install or run repo code (same rule as claude.yml review jobs).
- Concurrency group
claude-weekly-audit (not cancel-in-progress) so two scheduled
runs never overlap and double-file.
- Auth is the same OAuth-preferred / API-key-fallback wiring as every
claude.yml job,
covered by claude-auth-canary.yml.
