Menu
Ingestion pipeline architecture overview and convention reference. Use when you need a quick orientation to the pipeline framework or want to know which doctor agent to use for a specific concern.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Required reading before writing any HogQL/SQL or calling execute-sql against PostHog. Use whenever the user wants to search, find, or do complex aggregations PostHog entities (insights, dashboards, cohorts, feature flags, experiments, surveys, hog flows, data warehouse, persons, etc.) and query analytics data (trends, funnels, retention, lifecycle, paths, stickiness, web analytics, error tracking, logs, sessions, LLM traces). Also the first stop for a governed business number (MRR, activation, revenue): check the semantic layer (canonical metrics in system.information_schema.metrics) for an approved definition before deriving from raw events. Covers HogQL syntax differences from ClickHouse SQL, system table schemas (system.*), available functions, query examples, and the schema-discovery workflow.
Debug, support, and build PostHog MCP Analytics โ product analytics for MCP servers (the `@posthog/mcp` and `posthog.mcp` SDKs plus the mcp_analytics product). Use when MCP analytics data looks wrong or missing ("events aren't showing", "intent clusters are empty", "sessions are missing", "per-tool numbers look wrong"), when writing queries over `$mcp_*` events by hand, or when doing feature work on the SDKs, the dashboard and its query runners, the self-instrumented MCP server, the `wizard mcp-analytics` install command, or the in-app onboarding. Covers the repo map, the `$mcp_*` vocabulary and where each property comes from, the rules that silently corrupt metrics when ignored, the end-to-end pipeline and where each stage breaks, and which repo to change. For reading the data rather than fixing it, prefer the `exploring-mcp-*` and `improving-mcp-tools` skills.
Edit the @posthog/quill design system locally and consume the change in products/desktop before it is published to npm. Use when changing quill components/primitives/tokens, when a quill change must be tested inside the Code app, or when the user mentions quill, the design system, the .local-quill tarball, or the @posthog/quill pnpm override.
How to add a new ingestion warning type to the event ingestion pipeline. Use when emitting a new warning from nodejs ingestion code (emitIngestionWarning, captureIngestionWarning, pipeline `warnings` arrays, `drop()` with warnings), when adding a warning type, category, or severity, or when a typecheck error says a string is not assignable to IngestionWarningType. Covers the INGESTION_WARNING_TYPES registry (the single source of truth for type, category, and severity), the details-key conventions that ClickHouse v2 materializes into columns, debouncing, and the downstream surfaces to keep in sync (v1 UI map, resolving-ingestion-warnings skill, docs, v2 API).
Diagnoses and resolves PostHog ingestion warnings โ problems recorded while ingesting events (dropped events, rejected person merges, oversized payloads, invalid data). Use when a user asks why events are missing, dropped, or undercounted, why identify/alias calls don't work or accounts stay duplicated, why person or group properties aren't updating or profiles look inflated, why recordings have gaps, heatmaps are empty, LLM token counts are missing, why cookieless events vanish, or whenever the `ingestion_warning` health check fires. Explains severity triage (error = dropped, warning = modified, info = intentional) and routes all warning types โ size limits and enrichment, merges and distinct IDs, `$process_person_profile`, timestamps, cookieless, heatmaps, transformations, session replay โ to per-issue reference files with code-level causes and per-SDK fixes.
Create or edit a PostHog canvas โ a sandboxed browser application (data board, document, form, small tool, graphics experiment) stored in PostHog and rendered by the desktop/web app. Use when a task asks to build, generate, update, or fix a canvas, or when a canvas id is given as the publish target. Covers resolving or creating the target canvas, choosing an implementation approach (React + Quill vs plain HTML/browser APIs), the read โ edit โ validate โ publish โ build loop, and which companion canvas skills to load for the details.
| name | ingestion-pipeline-doctor-nodejs |
| description | Ingestion pipeline architecture overview and convention reference. Use when you need a quick orientation to the pipeline framework or want to know which doctor agent to use for a specific concern. |
Quick reference for PostHog's ingestion pipeline framework and its convention-checking agents.
The ingestion pipeline processes events through a typed, composable step chain:
Kafka message
โ messageAware()
โ parse headers/body
โ sequentially() for preprocessing
โ filterMap() to enrich context (e.g., team lookup)
โ teamAware()
โ concurrentlyPerGroup(token:distinctId) for per-entity processing
โ gather()
โ pipeChunk() for chunk operations
โ handleIngestionWarnings()
โ handleResults()
โ handleSideEffects()
โ build()
See nodejs/src/ingestion/pipelines/analytics/joined-ingestion-pipeline.ts for the real implementation.
| What | Where |
|---|---|
| Step type | nodejs/src/ingestion/framework/steps.ts |
| Result types | nodejs/src/ingestion/framework/results.ts |
| Doc-test chapters | nodejs/src/ingestion/framework/docs/*.test.ts |
| Joined pipeline | nodejs/src/ingestion/pipelines/analytics/joined-ingestion-pipeline.ts |
| Doctor agents | .claude/agents/ingestion/ |
| Test helpers | nodejs/src/ingestion/framework/docs/helpers.ts |
| Concern | Agent | When to use |
|---|---|---|
| Step structure | pipeline-step-doctor | Factory pattern, type extension, config injection, naming |
| Result handling | pipeline-result-doctor | ok/dlq/drop/redirect, side effects, ingestion warnings |
| Composition | pipeline-composition-doctor | Builder chain, concurrency, grouping, branching, retries |
| Testing | pipeline-testing-doctor | Test helpers, assertions, fake timers, doc-test style |
Steps: Factory function returning a named inner function. Generic <T extends Input> for type extension. No any. Config via closure.
Results: Use ok(), dlq(), drop(), redirect() constructors. Side effects as promises in ok(value, [effects]). Warnings as third parameter.
Composition: messageAware wraps the pipeline. handleResults inside messageAware. handleSideEffects after. concurrentlyPerGroup for per-entity work. gather before chunk steps.
Batching lifecycle hooks (BatchingPipeline beforeBatch/afterBatch): enrich-only. Hooks may enrich elements and batch context but must return exactly the elements they received โ a count change is a broken invariant and feed() throws. Filtering belongs in sub-pipeline steps that return drop(). An empty feed() is a no-op (no hooks, no capacity). Details: nodejs/src/ingestion/framework/docs/14-batching.test.ts.
Fan-out/fan-in (fanOut(fn).via((sub) => โฆ).fanIn(fn)): per-element sub-work with cardinality restored โ one element fans out to N sub-elements (e.g. per-blob uploads), a regular sub-pipeline processes them (maxConcurrency on the sub concurrently block, retry on the per-sub step), and fan-in folds the OK results back into the parent. Reach for it over concurrently/concurrentlyPerGroup when the unit of concurrency is smaller than the element; hand-rolled p-limit/Promise.all inside a step is the tell. Sequencing is compile-time enforced (an unclosed stage cannot build). Sub-result contract: OK collected; DROP excludes the sub silently; DLQ fails the parent with aggregated reasons; REDIRECT is excluded with a warning โ sub redirects never escape the stage. Sub-pipelines are context-agnostic: team/message data goes in the sub-element value, and context-gated surface (teamAware, handleIngestionWarnings, โฆ) is uncallable. Fan-out/fan-in functions are cheap, synchronous, and named. Parents emit unordered as they complete. Details: nodejs/src/ingestion/framework/docs/17-fan-out-fan-in.test.ts.
Testing: Step tests call factory directly. Use consumeAll()/collectChunks() helpers. Fake timers for async. Type guards for result assertions. No any.
Ask Claude to "run all pipeline doctors on my recent changes" to get a comprehensive review across all 4 concern areas.