feature-usage-feed

# Building a feature usage feed via LLM evals Some PostHog features (group session summaries, single session summaries, replay AI search, error tracking AI debug, etc.) generate hundreds or thousands of LLM traces per week. Reading them by hand is not feasible. This skill covers the end-to-end pattern for turning that trace volume into a live Slack feed of canonical use cases — what users are actually doing with the feature. The workflow is **mixed, and leans UI**. Trace inspection and filter discovery (steps 1-2) are MCP-driven. Eval creation, dry-running, and enabling (steps 4-5) are MCP-driven _when_ `posthog:llma-evaluation-*` tools are exposed to your agent — but they often aren't, in which case fall back to the UI (Data pipeline → destinations for the alert is always UI). Each step flags its UI fallback. Expect to finish in the UI even when you start from chat. ## When to use - "How are people actually using [feature X] in production?" - "Can we identify the canonical use cases for [feature X] so we can write better docs / prioritize improvements?" - "I want a Slack feed of representative usage examples without manually skimming traces." - "Set up a feed of use cases for [feature X] in #team-[area]-usage." If the user just wants to debug a single trace or tune an existing eval, redirect to `exploring-llm-traces` or `exploring-llm-evaluations` instead. ## Two filter patterns This skill supports two different ways to scope an eval to "the feature you care about": **Pattern A — Feature-native trace_id prefix.** For standalone features that emit their own `$ai_trace_id` pattern (e.g. `session-summary:group:`, `replay-search:`, error-tracking-specific flows). Filter on the prefix. **Pattern B — PostHog AI agent mode.** For features the user interacts with _via_ PostHog AI in a specific agent mode (error tracking, product analytics, session replay, SQL, flags, surveys, AI observability). Filter on `ai_product = 'posthog_ai' AND agent_mode = '<mode>'`. This requires PR #55160 (merged April 2026) to be deployed, which threads `agent_mode` and `supermode` onto every `$ai_generation` emitted by the chat agent loop. A useful ergonomic side-effect: `agent_mode IS NOT NULL` is a reliable "user-facing chat turn" filter — batch jobs and tool-internal LLM calls go through different code paths and have `agent_mode=null`, so they're excluded for free. If the user asks "what are users trying to DO in [ET / replay / SQL / flags / surveys] mode of PostHog AI", that's Pattern B. If they ask "what use cases does [standalone feature] cover", that's Pattern A. Pick the pattern first — the prompt, filter, and Slack channel naming all follow from it. ## Prerequisites | Requirement | How to verify | | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | (Pattern A) Feature emits `$ai_generation` events with a stable `$ai_trace_id` pattern | `posthog:execute-sql` for distinct `$ai_trace_id` prefixes | | (Pattern B) `agent_mode` property is present on recent `$ai_generation` events | `posthog:execute-sql` group-by `properties.agent_mode` on recent `ai_product='posthog_ai'` events. Null bucket is normal (batch jobs + tool-internal calls) — you want non-null coverage across the modes you care about. | | `$session_id` is attached to the `$ai_generation` events (links trace to trigger session) | `posthog:execute-sql` for `countIf($session_id IS NOT NULL) / count()` | | `$session_id` is also attached to the `$ai_evaluation` events (lets the Slack alert link to the session) | Same query but on `$ai_evaluation` events after the eval has run once | | User has organisation-level AI data processing approval | Required for `llm_judge` evaluations | If `$session_id` is missing on either event type, file a backend fix before continuing — there is no UI workaround. The session-summary feature has a worked example of the threading pattern in PR #54952. For Pattern B, the agent-mode threading pattern is in PR #55160. ## Tools | Tool | Purpose | | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `posthog:query-llm-traces-list` | Find sample traces matching the feature's `$ai_trace_id` pattern | | `posthog:query-llm-trace` | Inspect a specific trace's contents end-to-end | | `posthog:execute-sql` | Verify trace volume, session_id coverage, eval result distributions | | `posthog:llma-evaluation-create` | (**often unexposed** — UI fallback: AI observability → Evaluations → New) Create the LLM-judge eval (disabled at first) | | `posthog:llma-evaluation-run` | (**often unexposed** — UI fallback: the eval's detail page has a "Run on event" button) Dry-run the eval against specific generations during prompt iteration | | `posthog:llma-evaluation-update` | (**often unexposed** — UI fallback: edit the eval in AI observability → Evaluations) Tweak the prompt / enable when ready | | `posthog:llma-evaluation-report-create` | (**often unexposed** — UI fallback: the eval detail page has a "Reports" tab) After the feed is running, schedule an AI report on the eval to keep watching signal quality | | `posthog:workflows-list` / `posthog:workflows-get` | (**often unexposed** — UI: Data pipeline → Workflows) Browse existing workflow configs — useful for cloning an existing feed's structure when setting up a new one. Read-only; no create/update tool is exposed yet, so step 6's Slack workflow setup is UI-only. | Before starting, **check which of the `posthog:llma-evaluation-*` tools are actually exposed in your agent's MCP tool set.** If they aren't loaded, treat steps 4-5 as UI walkthroughs rather than tool calls. ## Workflow ### Step 1 — Identify the filter **Pattern A (feature-native trace_id prefix):** find the prefix that maps to your feature. ```sql SELECT splitByChar(':', coalesce(properties.$ai_trace_id, ''))[1] AS root, splitByChar(':', coalesce(properties.$ai_trace_id, ''))[2] AS subtype, count() AS events FROM events WHERE timestamp > now() - INTERVAL 3 DAY AND event = '$ai_generation' AND properties.$ai_trace_id IS NOT NULL GROUP BY root, subtype ORDER BY events DESC LIMIT 25 ``` Note: `coalesce(..., '')` is load-bearing — `splitByChar` on a nullable column errors out in HogQL otherwise. **Pattern B (PostHog AI agent mode):** verify coverage and volume for the mode you're targeting. ```sql SELECT properties.agent_mode AS agent_mode, properties.supermode AS supermode, count() AS events, count(DISTINCT properties.$ai_trace_id) AS traces FROM events WHERE timestamp > now() - INTERVAL 3 DAY AND event = '$ai_generation' AND properties.ai_product = 'posthog_ai' GROUP BY agent_mode, supermode ORDER BY events DESC LIMIT 20 ``` Expected values for `agent_mode`: `error_tracking`, `product_analytics`, `sql`, `session_replay`, `flags`, `survey`, `llm_analytics`, `null`. Null ≈ batch jobs + tool-internal calls (not user chat). `supermode='plan'` splits planning turns from execution turns — worth calling out separately if your feed is about plan-mode specifically. Record the mode + rough volume. Low-volume modes (<100 events/day) will produce a trickle-feed that's hard to validate early; high-volume modes (>1k/day) may need sampling to avoid Slack flooding. See the "Tips" section on sampling. ### Step 2 — Pull a handful of sample traces Use these for prompt iteration in step 4. **Pattern A:** ```json posthog:query-llm-traces-list { "properties": [ { "type": "event", "key": "$ai_trace_id", "operator": "icontains", "value": "<your-prefix-here>" } ], "limit": 10, "dateRange": { "date_from": "-2d" }, "randomOrder": true } ``` **Pattern B:** ```json posthog:query-llm-traces-list { "properties": [ { "type": "event", "key": "ai_product", "operator": "exact", "value": "posthog_ai" }, { "type": "event", "key": "agent_mode", "operator": "exact", "value": "<mode-here>" } ], "limit": 10, "dateRange": { "date_from": "-2d" }, "randomOrder": true } ``` `randomOrder: true` matters — recency bias produces a non-representative sample. Pick 5-10 traces to test against. **Output size warning:** `query-llm-traces-list` with `limit: 10` routinely returns 3-6MB of JSON (full input/output per generation). This will blow your context window. **Immediately delegate the summarization to a subagent** the moment you see the "result exceeds maximum allowed tokens" error — ask the subagent to extract, per trace: the trace id, the first user message (truncated to ~300 chars), and a one-sentence description of what the conversation was about. Don't try to read the raw file in-line. The trace tools return the AI payload only. A custom property such as `$current_url`, `agent_mode` or `ai_product` comes back as a name in `_redactedKeys` with no value, so read it with `posthog:execute-sql` for the sampled trace ids when a sample needs it. Filtering on those properties, as both patterns above do, is unaffected. **Watch for topic drift in Pattern B samples.** The `agent_mode` tag reflects the user's mode selection at the time of the turn — but chat state retains the mode even if the user drifts off-topic within the same conversation (e.g. user selected "error tracking" mode, then asked an unrelated pricing question three turns later). Your eval prompt's classification step needs to be permissive about topic-drift: PASS should mean "user is doing something recognizably in-scope for this mode", FAIL should catch the off-topic drift. If you don't, your feed will include irrelevant PASS entries that happen to carry the mode tag. ### Step 3 — Draft the LLM-judge prompt The prompt has two responsibilities: (a) classify the trace as relevant or not, (b) produce reasoning text that is **directly postable to Slack** (no preamble, no meta-description). The reasoning field becomes the Slack message body. Template: ```text You are analyzing a PostHog [FEATURE NAME] trace to extract its real use case. Your reasoning text will be posted directly to a Slack channel as a notification. Write it as a short, ready-to-post message — no preamble, no meta-description. Step 1 — Classification: - PASS = this trace is the [feature kind] you care about - FAIL = a different LLM call or a false match - N/A = ambiguous from the trace alone Step 2 — Reasoning (only matters if PASS). Write 2-3 sentences in this exact format: "[OPENER] [what they targeted/filtered for]. They were trying to [understand X / debug Y / find Z]. The result surfaced [key pattern or finding]." Your output MUST start with the exact phrase "[OPENER]". No other opening is allowed. Rules: - No "This is a [feature]..." or "The input contains..." preamble - No JSON, field names, system-prompt references, or meta-description - Concrete > generic. "users hitting error tracking for the first time" beats "user behavior" - If you cannot infer one of the three pieces from the trace, write "(unclear from trace)" in that slot — do not guess ``` **Pick an `[OPENER]` that matches how users actually interact with the feature.** The forced opener is load-bearing (it prevents the model from drifting into "this trace is a..." meta-description), but the exact verb has to fit the interaction: | Feature / mode | OPENER | | --------------------------------- | ------------------------------------------ | | Session summary (group / single) | `A user ran a summary on` | | Replay AI search | `A user searched replays for` | | PostHog AI in error tracking mode | `A user asked PostHog AI about` |