relay-conformance-audit

# Cocoa ↔ Relay conformance audit The SDK hard-codes many strings and wire formats that must match Relay exactly — header names, envelope item types, data categories, discard reasons, DSC keys, session fields. A mismatch anywhere in that surface fails **silently**: data is dropped, mis-routed, or miscounted with no error. This skill audits that entire surface. One motivating example of the class: [#8322](https://github.com/getsentry/sentry-cocoa/issues/8322) (a case-sensitive `X-Sentry-Rate-Limits` header read silently rate-limited all telemetry) — but the skill exists to hunt bugs _like_ it, wherever they occur, not that one bug. **READ-ONLY toward the SDK.** Never edit SDK code. Never modify `references/findings.md` (the human-maintained ignore list). The skill's primary output is **`relay-audit:` GitHub issues** (created or edited — see "GitHub issues"). The one other permitted write is a draft PR the coverage check opens against `getsentry/sentry-cocoa` that touches nothing but `references/surface-map.md` (see "Coverage check"). When the map is out of date, the skill opens that draft PR directly rather than only flagging the gap. ## Model Every run evaluates the **full current list of mismatches** — no delta tracking, no run-to-run memory. GitHub is the tracker; `references/findings.md` holds only the ignore list. A mismatch is in one of three states: - **Ignored** — listed in `references/findings.md` with an explicit _ignore-scenario_. Skipped (counted only) while the scenario holds; becomes needs-action when it no longer holds. - **Needs action** — everything else, unless cross-SDK corroboration drops it. Matched to a GitHub issue by fingerprint and filed via the pipeline in "GitHub issues" (open match → edit; expired-ignore closed match whose back-ref proves the audit closed it → reopen; other closed match → comment once; no match → create). - **Cross-SDK false positive** — a needs-action mismatch whose `all-sdks-agree` verdict drops it: no issue, no per-finding detail (so a false positive never reads as real), counted only as an aggregate `dropped all-sdks-agree <n>` in the console summary (see "Cross-SDK corroboration"). A live GitHub search precedes every create, so the audit never opens a duplicate. It also never closes issues or edits the ignore list itself — the reverse checks (Procedure step 6) only surface stale open issues and stale ignore entries for a human to act on. ## Procedure 1. Record `git rev-parse HEAD` (full SHA — needed for permalinks), `git rev-parse --short HEAD`, and `date +%Y-%m-%d` via shell (never hardcode); the full SHA builds the code links, the short SHA + date stamp each issue. 2. For each area in `references/surface-map.md`, spawn one read-only subagent (parallel). Contract: fetch the listed Relay source / develop-docs pages, read the listed Cocoa files, diff per the area's notes. Return mismatches as `{area, severity HIGH|MEDIUM|LOW, file + symbol, line range (start–end), exact wire strings, spec citation, failure mode}`. The **line range is required** — it builds the commit-pinned permalink. Severity = silent blast radius, not fix effort. In parallel with these, spawn the coverage-check subagent (see "Coverage check"). 3. Classify each mismatch against `references/findings.md` on fingerprint (`area + file + normalized summary` — never line numbers/severity; those vary between runs): listed and ignore-scenario still holds → **ignored**, skip; listed but scenario **no longer holds** → **needs action (expired-ignore)**, carrying that flag and, if present, the entry's `(relay-audit #<n>)` back-reference; not listed → **needs action**. Every expired-ignore finding also means its `findings.md` row is now stale — **always** surface that row in the console summary under `ignore-list entries expired — verify & prune` for a human to remove, independent of which issue path Dedup then takes (Dedup reopens only when the entry's back-ref matches the closed issue it hits; otherwise create / edit-open / comment-closed like any other finding). Without this, entries added before back-refs existed (or that take `create`/`open-edit`) would keep the finding permanently flagged `expired-ignore` and permanently exempt from the `all-sdks-agree` drop. 4. **Cross-SDK corroboration** (see below): for every mismatch that is still **needs action** after step 3, check how the peer SDKs implement the same wire surface. This is the confidence gate — it strengthens `cocoa-only` findings with exact peer code locations and silently drops `all-sdks-agree` false positives. **Exception: an `expired-ignore` finding (step 3) is never dropped here** — it still reproduces and its stale ignore row must be surfaced, so it always proceeds to step 5 regardless of verdict (its cross-SDK result is recorded in whichever issue Dedup files/reopens/comments). 5. For each mismatch that survives the cross-SDK gate, run the GitHub-issue pipeline (see "GitHub issues" → Dedup) to file it. 6. **Reverse checks (no-longer-reproduces).** Two symmetric checks against this run's **raw mismatch set from Procedure step 2** (every mismatch observed — reported, ignored at step 3, and dropped as `all-sdks-agree` at the cross-SDK gate; all still reproduce, only the pipeline's disposition differs — so match on the raw set, not the filed subset; enumerating a subset and forgetting a category is exactly what mis-flags): - **6a — open issues.** List all **open** `relay-audit:` issues (`gh issue list --repo getsentry/sentry-cocoa --search "relay-audit: in:title" --state open --limit 1000 --json number,title,body`; `body` is required to reconstruct each issue's fingerprint — `file` lives in the body, so without `--json ...,body` the match can't be built; same pagination guard as dedup — if the result equals the limit, page before trusting it). Any open issue with **no** matching mismatch this run is a no-longer-reproduces candidate: **do not close it** (a run may miss a finding for benign reasons — a moved file, a flaky peer search); list it in the console summary under "no longer reproduces — verify & close". This backs the Model's promise; without it a fixed mismatch's issue would linger forever. - **6b — ignore-list entries.** Diff each `references/findings.md` ignored entry's fingerprint against the same raw mismatch set. Any ignored entry with **no** matching mismatch this run no longer reproduces: **do not edit `findings.md`** (read-only toward it); list its fingerprint in the console summary under "ignore-list entries no longer reproducing — verify & prune" so a human prunes it. This backs the Model's promise for the ignore list, symmetric to 6a for issues. 7. Print the console summary (see "Console summary") and append the coverage-check result. If `gh` is unauthenticated or any `gh` call fails, print everything and state loudly that the issue write FAILED — never report success on a failed write. ## Coverage check (surface-map self-audit) The audit only inspects what the surface map lists, so a stale map is a silent coverage hole. Every run, one extra subagent checks the map itself — from both directions: - **SDK side:** list protocol-relevant code the map does NOT reference. Grep the SDK for wire-emitting surfaces: `serialize` implementations feeding envelopes, envelope item construction, `setValue(_:forHTTPHeaderField:)` / header reads, files under `Sources/Swift/Networking/` and `Sources/Swift/Protocol/`, new integrations writing envelope items. Compare against the files the map lists; report unlisted ones that put bytes on the wire. - **Relay side:** for each Relay/spec source the map cites, check it still exists (note moved paths) and skim Relay's protocol enums (`DataCategory`, `ItemType`, discard reasons, DSC keys) for members added since the map's checks were written that no area would catch. - Ignore non-wire code (`SentryCrash/` internals, UI, tests) — the bar is "could a mismatch here silently corrupt what Relay receives?". Outcome: - **Map up to date** → one line in the console summary: `coverage: OK`. - **Gaps found** → list them in the console summary under `coverage: GAPS`, **and directly open a draft PR** against `getsentry/sentry-cocoa` (base `main`) that updates ONLY `references/surface-map.md` — don't just describe it, push it. The map drifts as the SDK changes, so this is the mechanism that keeps it current. Mechanically: 1. Edit `skills/relay-conformance-audit/references/surface-map.md` only — extend an area's file list, fix a moved Relay path, or add a new area (with **What it is / Cocoa / Relay-Spec / Check**, all links as `blob/main` (Cocoa) or `blob/master` (Relay), matching the file's existing format). 2. Branch `chore/relay-audit-surface-map-<date>`. Commit just that file: `chore: update relay-audit surface map` with `#skip-changelog` in the body. 3. Push and `gh pr create --draft --base main --title "chore: update relay-audit surface map"`; body = the gap list + one line per gap on why it belongs in the map + "Opened by relay-conformance-audit coverage check, <DATE>, @<SHA>." Put the resulting PR URL in the console summary's `coverage` line. - This is the skill's **only** permitted repository write (issue creation/editing aside): never touch SDK code, `findings.md`, or `SKILL.md` in that PR; leave it as a draft and never merge — a human reviews. If a draft PR from a previous run is still open, push to that branch and update its body instead of opening a second one. If `gh` isn't authenticated or the push fails, say so loudly in the console summary and include the diff inline — never report the PR as opened when it wasn't. - Gaps are NOT mismatches: list them in the coverage section only; the newly-discovered files get audited by the next run after the map PR merges. ## Cross-SDK corroboration Diffing Cocoa against Relay + develop-docs alone produces false positives: when the develop-docs spec is aspirational/wrong, or Relay tolerates a value, the docs-vs-Relay disagreement gets blamed on Cocoa. The peer SDKs are a second source of truth. If Cocoa is the **only** SDK that diverges, the finding is almost certainly real; if **every** SDK does the same thing Cocoa does, the "mismatch" is almost certainly a misreading of the spec, not a Cocoa bug. Run this only on mismatches that are still **needs action** after the `findings.md` match (step 3) — never on ignored entries. That bounds the extra searches to exactly the findings that would be reported. Peer repos, in priority order (mobile first): `getsentry/sentry-java`, `getsentry/sentry-dart`, `getsentry/sentry-react-native`, then `getsentry/sentry-javascript`, `getsentry/sentry-python`. **Locate peer code by the wire string, not a file map.** The exact literal the mismatch is about (discard-reason string, header name, envelope item-type, DSC key, session field) is the stable locator — search each repo for it, e.g. `gh search code --repo getsentry/sentry-java "ratelimit_backoff"`. This lands directly on the peer's implementation with no per-SDK map to maintain and keep in sync. If the literal search misses (a repo names the constant differently), fall back to the role-analogous file, then move on — a peer you genuinely can't locate is recorded as not-checked, not as agreement. **Count only independent implementations.** A peer only corroborates if it implements the audited surface **itself**. When a peer is a wrapper that delegates the surface to another SDK already in the list, its result is inherited, not independent evidence — so record it as `not-checked` and exclude it from the quorum. In practice this is `sentry-react-native`: for native wire surfaces (transport, rate-limit headers, envelope framing) it delegates to the native `sentry-cocoa`/`sentry-java` SDKs, and for JS-level surfaces it reuses `@sentry/core` from `sentry-javascript` — in both cases the underlying implementation is another peer you already check, so counting RN too would let a **single** implementation satisfy the two-peer quorum. RN counts only when it carries its **own** implementation of the specific surface. When unsure whether a peer is independent for a given surface, treat it as `not-checked` — conservative, never inflates the quorum. For each needs-action mismatch, classify each **independent, located** peer into exactly one of three buckets — **matches Relay** (implements the spec-correct value Cocoa is missing), **matches Cocoa** (shares Cocoa's divergent value), or **matches-neither** (implements a _third_ value/format that is neither Relay's nor Cocoa's). A peer you couldn't locate, or a non-independent wrapper, is `not-checked` and counts toward none of the three. Then set a verdict. The five verdicts are mutually exclusive; evaluate them in this order: - **`inconclusive`** — fewer than **2** independent peers could be located (mobile SDKs preferred in the count). Corroboration is too thin to trust either way → **report as needs-action** regardless, flagged _"cross-SDK inconclusive — only <n> independent peer(s) located"_. This case takes precedence over the others: never silently drop a finding on weak evidence, because a wire-string search that misses everywhere would otherwise let a real bug vanish. This is the guard against a vacuous `all-sdks-agree`. - **`ecosystem-divergent`** — ≥2 independent peers located and **at least one is matches-neither** → a third implementation exists, which is itself a signal of spec ambiguity, not Cocoa-specific breakage. **Always report** as needs-action; call out each third value and its peer. A `matches-neither` peer can never be folded into agreement, so this blocks both `cocoa-only` and `all-sdks-agree`. Takes precedence over the three below. - **`cocoa-only`** — ≥2 independent peers located and **every one matches Relay** (none match Cocoa, none matches-neither) → Cocoa is the sole outlier, finding **confirmed and strengthened**. Report it, flagged higher-confidence, with clickable `blob/`-pinned links to the peer code showing the correct implementation. - **`mixed`** — ≥2 independent peers located, all classify as either matches-Relay or matches-Cocoa (no matches-neither), and they **split** (at least one each) → report normally; note which peers side with Cocoa and which with Relay. Do not label a split `cocoa-only`. - **`all-sdks-agree`** — ≥2 independent peers located and **every one matches Cocoa** (none matches Relay, none matches-neither) → strong **false-positive** signal. Before acting, re-read the Relay source directly to confirm what it actually does with the value. Default: **drop the finding** — file no issue and emit no per-finding detail; the only trace is the aggregate `dropped all-sdks-agree <n>` counter in the console summary (never the finding's identity). Escape hatch: if after re-reading Relay you are still certain it is a real wire bug affecting all SDKs, report it as needs-action, explicitly labeled _"affects all SDKs — likely a spec/Relay-wide issue, not Cocoa-specific"_ and sorted last; an escape-hatched finding is filed as an issue and is NOT part of the dropped count. The escape hatch is certainty-gated, so the common case stays a silent count. Add the result to each reported mismatch as `crossSDK: { located:[repos], notChecked:[{repo, reason: unlocated|wrapper}], matchRelay:[{sdk, link, note}], matchCocoa:[{sdk, link}], matchNeither:[{sdk, link, value}], verdict: inconclusive|ecosystem-divergent|cocoa-only|mixed|all-sdks-agree }`. This block feeds the issue body's cross-SDK section. ## GitHub issues Each needs-action mismatch maps to exactly one `relay-audit:` GitHub issue on `getsentry/sentry-cocoa`. Dedup (below) is the authoritative branch — create / edit-open / reopen-expired-ignore / comment-closed. **Never open a duplicate; the only closed issue the audit reopens is one whose `(relay-audit #n)` back-ref proves the audit itself closed it — never a human-closed issue.** ### Dedup (do this before every create) 1. `gh issue list --repo getsentry/sentry-cocoa --search "relay-audit: in:title" --state all --limit 1000 --json number,title,body,state,comments,updatedAt` — open + closed. Each field is load-bearing: `state` (branch open/closed), `comments` (skip re-posting a closed-match comment), `updatedAt` (pick the newest when several match — default order is relevance, not recency), `body` (holds `file`, needed for the fingerprint). If the result equals `1000` the cap was hit — paginate before trusting a "no match". 2. Match the mismatch's fingerprint (`area + file + normalized summary`, never line numbers/severity) against each candidate's title + body. On multiple matches, **prefer the most-recently-updated open issue** (edit it); a closed match drives the closed path only when no open match exists. Note extra same-fingerprint duplicates in the console summary for a human to consolidate. 3. **Match found** → branch on the winning match's `state`: - **open** → `gh issue edit <number> --body-file <tmp>` (rebuilt body; keep the title unless area/summary changed). - **closed + expired-ignore, and the audit provably closed _this_ issue** — the `findings.md` entry carries a `(relay-audit #n)` back-ref **and** the matched closed issue's number equals `n` → **reopen** (the only case where the audit is certain it did the closing): `gh issue reopen`, `gh issue edit` (rebuilt body), one `gh issue comment` noting the scenario lapsed as of `<SHORT_SHA>` (<DATE>) and to remove the `findings.md` row. Summary: `reopened (ignore expired): #<n>`. - **closed, any other reason** — including an expired-ignore whose entry has **no** back-ref, or whose back-ref points at a _different_ issue than the one matched (so the audit can't prove it closed this one; a human may have) → possible regression, but don't presume why a human closed it. **Don't reopen/edit/create** — post one `gh issue comment` carrying the stable marker `<!-- relay-audit:reproduces-on-closed -->` (skip if any comment already has that marker — match the marker, not the SHA/date text). Always list it in the console summary under `closed but still reproduces — reopen or ignore: #<n>`, whether or not the comment was posted this run. 4. **No match** → create in two steps (the ignore prompt needs the issue's own number): (a) `gh issue create --title … --label relay-audit --body-file <tmp>` with the literal `#<ISSUE_NUMBER>` placeholder, capture the new number; (b) `sed` the real number into every `<ISSUE_NUMBER>` and `gh issue edit`; verify none remain. ### Title `relay-audit: <area> — <one-line summary>` — the `relay-audit:` prefix is **mandatory** on every issue. ### Labels `relay-audit` (lowercase). If the label doesn't exist yet, create it first (`gh label create relay-audit --description "Cocoa ↔ Relay wire-format conformance audit" --color BFD4F2`), then pass `--label relay-audit` on create. ### Body (identical shape for create and edit) First an **automated-origin banner** (a blockquote — so anyone landing on the issue immediately knows a bot filed it and how), then a short human intro that **links the exact code inline**, then the full agent-pickup analysis inside a collapsed `<details>` block. Permalinks use the **full SHA** recorded in step 1 of the Procedure so they pin to the audited commit and stay clickable: The body below is shown in a tilde-fenced (`~~~`) example so the inner triple-backtick prompt block nests cleanly — the issue itself uses normal triple-backtick fences. ````markdown > 🤖 Automated issue from the [`relay-conformance-audit`](https://github.com/getsentry/sentry-cocoa/blob/<FULL_SHA>/skills/relay-conformance-audit/SKILL.md) skill — weekly bot audit, not hand-filed. Run <DATE> @ `<SHORT_SHA>`. False positive? Use the 🔕 prompt below. --- <1–3 plain sentences for a human deciding whether to care: what silently breaks, user impact, blast radius. Keep it jargon-free, BUT anchor each claim to the exact code with an inline commit-pinned permalink — e.g. "the SDK rounds the value before sending it ([`SentryTraceContext.m#L108`](https://github.com/getsentry/sentry-cocoa/blob/<FULL_SHA>/<path>#L108))". A human should be able to click straight to the offending line from the intro, not only from the details block.> **Proof of the correct way (peer SDK):** <always present this line; never omit it. For `mixed`/`cocoa-only`, link a Relay-matching peer; for `ecosystem-divergent`, link a third-value peer — an inline `blob/`-pinned link to the exact peer line, e.g. sentry-java at [`Baggage.java#L41`](https://github.com/getsentry/sentry-java/blob/<peer-sha>/...#L41). For the two verdicts with no peer to cite, state why instead of linking: `inconclusive` → "Proof: none — fewer than 2 independent peers located; evidence thin"; escape-hatched `all-sdks-agree` → "Proof: none — every peer matches Cocoa; escalated as a likely spec/Relay-wide issue" plus a link to the Relay source you re-read.> Cross-SDK verdict: <verdict + one-clause summary>. **Severity:** HIGH|MEDIUM|LOW · **Area:** <area> <details> <summary>Full analysis (for an agent to pick up)</summary>