surface-audit

name

surface-audit

description

Audit a product UI surface (nav item, dashboard card, panel) to rule keep / demote / kill using warehouse behavioral data, distinguishing nav position from real demand, then draft the P2. Use when the user says "audit this surface/dashboard/nav", "should we keep/kill X", "is X actually used", "run the surface audit", or "removal audit". Generalizes the MSD-removal-audit method: rank by demand not raw clicks, exempt rare-but-critical surfaces, verify every headline number twice before it ships in the P2.

Surface audit — keep / demote / kill

Core rule: arrivals/page-views measure WHERE a surface sits (nav position); a genuine action event measures DEMAND. Rank by demand, never by raw clicks.

1. Scope

• List every surface under audit: name + route/slug + nav slot.
• Freeze the analysis window (default last 90d) and write it down. Every number below is "over this window."

2. Name the real-action event per surface

Before pulling any number, write down two events per surface:

• arrival event (page view / menu click) — proves someone landed there.
• real-action event (a genuine deploy / monitor / save / config / publish) — proves demand. ⚠ Menu-click ≠ demand. A nav-click event measures navigation, not desire. Never let a click stand in for the real-action event. ⚠ 1:1 events are arrivals in disguise. If an event fires exactly once per page view, it measures arrivals — discard it as a demand signal.

3. Query Trino

Pull per surface from tracks.prod_events (+ related) via mcp__trino__execute_sql: arrivals, real-action count, arrivals→action ratio. Skeleton (adapt per surface — the real-action event differs each time):

SELECT
  count(*) FILTER (WHERE _en = '<arrival event>')        AS arrivals,
  count(*) FILTER (WHERE _en = '<-- replace: real-action event>') AS actions,
  count(DISTINCT user_id) FILTER (WHERE _en = '<-- replace: real-action event>') AS action_users
FROM tracks.prod_events
WHERE _dt >= date_add('day', -90, current_date)   -- match the frozen window
  AND path LIKE '<route/slug>%';

⚠ Always cite the SQL in the working notes, next to the number it produced.

4. Instrumentation-trap checklist — run before trusting any number

• ⚠ Double-slug bug: same surface logged under two slugs inflates unique counts. Reconcile slug variants before counting. If found → file a follow-up ticket.
• ⚠ Menu-click counted as demand: re-check that the "action" event isn't just a nav click (step 2).
• ⚠ 1:1 with page views: if action count ≈ arrivals for a surface, the event measures arrivals — drop it, find a truer signal.
• ⚠ Top-of-menu inflation: high clicks on a top-positioned item are usually pass-through. Do not read position-driven traffic as demand. Flag any suspect metric — a suspect number gets no ruling until fixed or replaced.

5. Fire-extinguisher exemption

⚠ Low use ≠ kill for rare-but-critical surfaces: security, recovery, backups, billing failure, safety. Tag these — exempt from demote/kill regardless of usage. Check this list before ruling on anything.

6. Rule — every surface resolves to one bucket + a number

• KEEP — true demand (the core loop + spine): real-action count justifies it.
• DEMOTE — position-not-demand: high arrivals, low arrivals→action ratio.
• FOLD / KILL — the tail: negligible action after the traps and exemption are cleared. Every demote/kill line carries the figure that justifies it as N of M + % over the window. ⚠ Absolute figures only. No bare percentages, no "most users."

7. Verify before it ships

⚠ No number ships unverified. For each headline figure, run a second query from a different angle — distinct users vs raw events, or a date-bounded recount. Only verified absolute figures go in the P2.

8. Draft the P2

Draft the P2 in your own voice, or your voice skill if you have one. Include:

• the ruling sentence: "arrivals measure nav position, engagement measures demand."
• the keep / demote / kill table, each demote/kill row carrying its N-of-M + % + window.
• a follow-up instrumentation ticket list (e.g. fix the double-slug bug, add a clicks≠demand event) — reference the traps from step 4, don't restate them.