| name | posthog-analyst |
| description | Analyze pi agentic conversations, errors, and misalignments using PostHog MCP. Queries $ai_generation, $ai_span, $ai_trace events, segments by model, detects failure patterns, cross-references the wiki's model-adaptive harness research, and generates model-specific recommendations filed to the wiki. Use this skill when the user wants to analyze agent performance, understand where the harness breaks per model, audit LLM errors or tool failures, improve model profiles based on real data, or run a postmortem on agentic sessions. Trigger on: "posthog analyze", "/posthog-analyze", "analyze agent sessions", "audit llm errors", "postmortem", "what's failing per model", "harness performance analysis", "find misalignments", "model error patterns", "agent conversation audit".
|
| allowed-tools | Read Write Bash Glob Grep |
| compatibility | {"requires":["PostHog MCP bridge enabled in pi","POSTHOG_PERSONAL_API_KEY set"]} |
posthog-analyst: Agentic Conversation Analysis via PostHog MCP
You analyze real agentic conversations captured in PostHog to find where the harness breaks — per model. You do NOT auto-fix. You generate specific, actionable recommendations and file them to the wiki as structured analysis pages. The wiki's model-adaptive harness research is your reference for what failure modes to look for and what fixes exist.
Core Loop
1. VERIFY MCP → /posthog-mcp-status
2. QUERY EVENTS → execute-sql against $ai_generation, $ai_span, $ai_trace
3. SEGMENT → Group by $ai_model, $ai_agent_name, $ai_stop_reason
4. DETECT → Match patterns against known model failure modes (from wiki)
5. CROSS-REF → Read wiki model-adaptive-harness pages for fix catalog
6. RECOMMEND → Generate per-model recommendations
7. FILE → Write findings to vault/wiki/analyses/posthog-YYYY-MM-DD.md
Every analysis gets a wiki page. Knowledge compounds.
Step 1: Verify PostHog MCP
Run /posthog-mcp-status to confirm the bridge is connected. If it fails, tell the user:
PostHog MCP not connected. Set POSTHOG_PERSONAL_API_KEY and restart pi, or run /posthog-mcp-reload.
If the user hasn't set up PostHog tracing (no $ai_* events being captured), tell them:
No LLM analytics being captured. Set POSTHOG_API_KEY to your project key and restart pi. Events will start flowing immediately.
Step 2: Query Events
Use the PostHog MCP execute-sql tool (or equivalent query tool registered by the bridge). Query the following event types, filtered to this project:
Core Queries
A) Error rate by model — $ai_generation events:
SELECT
properties.$ai_model as model,
countIf(properties.$ai_is_error = true) as errors,
count() as total,
round(errors / total * 100, 1) as error_pct,
avg(properties.$ai_latency) as avg_latency_sec,
avg(properties.$ai_total_tokens) as avg_tokens
FROM events
WHERE event = '$ai_generation'
AND properties.$ai_project_name = 'ultimate-pi'
AND timestamp >= now() - INTERVAL {days} DAY
GROUP BY model
ORDER BY total DESC
B) Tool failure rate by model — $ai_span events:
SELECT
properties.$ai_model as model,
properties.$ai_span_name as tool,
countIf(properties.$ai_is_error = true) as errors,
count() as total,
round(errors / total * 100, 1) as error_pct,
avg(properties.$ai_latency) as avg_latency_sec
FROM events
WHERE event = '$ai_span'
AND properties.$ai_project_name = 'ultimate-pi'
AND timestamp >= now() - INTERVAL {days} DAY
GROUP BY model, tool
HAVING errors > 0
ORDER BY error_pct DESC
C) Stop reason distribution by model — $ai_generation events:
SELECT
properties.$ai_model as model,
properties.$ai_stop_reason as stop_reason,
count() as count,
round(count() * 100.0 / sum(count()) OVER (PARTITION BY properties.$ai_model), 1) as pct
FROM events
WHERE event = '$ai_generation'
AND properties.$ai_project_name = 'ultimate-pi'
AND timestamp >= now() - INTERVAL {days} DAY
GROUP BY model, stop_reason
ORDER BY model, count DESC
D) Token/cost efficiency by model — $ai_trace events:
SELECT
properties.$ai_model as model,
count() as sessions,
avg(properties.$ai_total_input_tokens) as avg_input_tokens,
avg(properties.$ai_total_output_tokens) as avg_output_tokens,
avg(properties.$ai_latency) as avg_session_sec,
countIf(properties.$ai_is_error = true) as errored_sessions
FROM events
WHERE event = '$ai_trace'
AND properties.$ai_project_name = 'ultimate-pi'
AND timestamp >= now() - INTERVAL {days} DAY
GROUP BY model
ORDER BY sessions DESC
E) High-error traces — raw error messages by model:
SELECT
properties.$ai_model as model,
properties.$ai_error as error_message,
properties.$ai_user_prompt as user_prompt,
count() as occurrences,
min(timestamp) as first_seen,
max(timestamp) as last_seen
FROM events
WHERE event IN ('$ai_generation', '$ai_trace')
AND properties.$ai_project_name = 'ultimate-pi'
AND properties.$ai_is_error = true
AND timestamp >= now() - INTERVAL {days} DAY
GROUP BY model, error_message, user_prompt
ORDER BY occurrences DESC
LIMIT 50
Date Range
If the user specifies a date range, use it. Otherwise default to the last 7 days. If they say "all time" or "everything", use 30 days.
If queries return empty: the project may not have tracing active yet. Tell the user and stop.
Step 3: Segment and Detect Patterns
Group data by model. For each model present, run through this checklist:
Pattern 1: Premature Completion (GPT Warning)
Signals: High stop_reason=stop rate without tool calls, low tokens/turn, sessions end after 1-2 turns
Wiki reference: [[harness-configuration-layers]] — L2 Gate Design (GPT needs hard-gate, L4 Completion Model)
What it looks like in data: Model says "I'm done" after plausible-but-incomplete answer. Stops before verification gate.
Pattern 2: Tool Loop (any model)
Signals: Same tool called 3+ times in one trace, tool returns errors but agent retries same approach
Wiki reference: [[execution-feedback-loop]] — agent doesn't change strategy on failure
What it looks like in data: Consecutive $ai_span events with same tool name, same or similar input, $ai_is_error=true on multiple consecutive spans.
Pattern 3: High Token Waste (any model)
Signals: Sessions with >2x median tokens, high cache-miss rates, large output tokens relative to task complexity
Wiki reference: [[ast-truncation]], [[progressive-disclosure-agents]]
What it looks like in data: $ai_total_tokens outliers, cache_read_input_tokens near zero despite repeated reads.
Pattern 4: Model-Specific Misalignment
Signals by model (cross-reference wiki [[harness-configuration-layers]] for dimensions):
| Model | Failure Mode | Data Signal | L1-L4 Layer |
|---|
| GPT variants | Stops early (premature completion) | Stop reason stop without verification gate passing | L2 Gate, L4 Completion |
| GPT variants | Anchors on first-seen content, ignores updates | Low tool retry rate despite errors — agent doubles down | L1 Signal (constraints-first needed) |
| GPT variants | Confused by nested structures | High error rate on multi-step tasks, tool chains break | L1 Signal (flat structure needed) |
| Claude/Opus | Overconfident self-assessment | Stop reason stop, no errors, but output misses spec requirements | L2 Gate (needs checklist, not self-assessment) |
| Claude/Opus | Ignores explicit markers if too verbose | Long output, many tokens, but key constraints missed | L1 Signal (too much narrative drowns signals) |
| Haiku (subagents) | Hallucinates in summaries | $ai_agent_name contains /explore or /haiku, error rate on read | L2 Gate (hard evidence gate needed) |
Pattern 5: Drift from Spec
Signals: Error messages contain "mismatch", "not found", "unexpected", "did not match". Sessions where later turns undo earlier edits.
Wiki reference: [[verification-drift-detection]], [[grounding-checkpoints]]
What it looks like in data: Sequential edit tool calls that revert each other, errors on verification steps.
Step 4: Cross-Reference Wiki Fix Catalog
For each detected pattern, read the relevant wiki page to find the fix catalog:
| Pattern | Wiki Page | Fix Type |
|---|
| Premature completion | [[harness-configuration-layers]] L2, L4 | Add hard gate, falsifiable checklist, enforced self-audit |
| Tool loops | [[execution-feedback-loop]] | Diff-aware testing, structured error output, max retry gates |
| Token waste | [[ast-truncation]], [[progressive-disclosure-agents]] | AST truncation, L0-L3 progressive disclosure, cache hints |
| Model-specific misalignment | [[model-adaptive-harness]], [[harness-configuration-layers]] | Adjust per-model dimension values in model-profiles |
| Drift from spec | [[verification-drift-detection]], [[grounding-checkpoints]] | Smallest-verifiable-change, re-grounding triggers |
| Haiku hallucination | [[model-routing-agents]] | Confidence scoring, critical reads stay on frontier |
| Repeated edit failures | [[fuzzy-edit-matching]], [[inline-post-edit-validation]] | Fuzzy matching, syntax validation before surfacing error |
Read these pages to extract the concrete fix options. Recommendations must reference specific wiki pages with wikilinks.
Step 5: Generate Recommendations
For each model with detected issues, produce recommendations in this format:
### Model: `$ai_model_value`
**Sessions analyzed**: N | **Error rate**: X% | **Avg tokens/session**: Y
#### Issue 1: [Pattern name]
- **Evidence**: [specific data — error counts, examples, rates]
- **Root cause**: [why this model fails this way, with wiki reference]
- **Recommendation**: [concrete action]
- **Layer affected**: L1/L2/L3/L4 (from harness configuration layers)
- **What to change**: [specific file, dimension, value]
- **Expected impact**: [what should improve]
#### Issue 2: ...
Recommendations are concrete and actionable. They reference specific files to change (e.g., .pi/skills/wiki-autoresearch/references/model-profiles.md, specific SKILL.md sections, harness config). They specify which harness layer and which dimension.
Do NOT auto-apply fixes. Only recommend. The user decides what to change.
Step 6: File to Wiki
Write the full analysis to vault/wiki/analyses/posthog-YYYY-MM-DD.md:
---
type: analysis
title: "PostHog Agent Analysis — YYYY-MM-DD"
created: YYYY-MM-DD
updated: YYYY-MM-DD
tags: [posthog, agent-analysis, model-performance, harness]
sources:
- "PostHog MCP — $ai_generation, $ai_span, $ai_trace events"
related:
- "[[model-adaptive-harness]]"
- "[[harness-configuration-layers]]"
status: complete
---
[2-3 sentence executive summary. Models analyzed, top finding, most critical recommendation.]
- Date range: [start] to [end]
- Events analyzed: N generations, M spans, T traces
- Models found: [list]
- Project filter: ultimate-pi
[Recommendations per the template above]
[Patterns that appear across multiple models — systemic harness issues, not model-specific]
| Priority | Model | Issue | Recommendation | Layer |
|----------|-------|-------|----------------|-------|
| Critical | ... | ... | ... | ... |
| High | ... | ... | ... | ... |
| Medium | ... | ... | ... | ... |
[What to do with these findings. Suggest a formal decision record update if recommendations are significant.]
After filing, update vault/wiki/index.md (add to analyses if category exists, or note inline), update vault/wiki/log.md (append entry at TOP), and update vault/wiki/hot.md (add key findings to Recent Context).
Quick Mode
If the user says "quick posthog analysis" or "just error rates per model", run only queries A and B, produce a summary table, and skip detailed recommendations. Offer to go deep if they want more.
## Quick Summary
| Model | Generations | Errors | Error % | Avg Tokens | Avg Latency |
|-------|-------------|--------|---------|------------|-------------|
| claude-sonnet-4-20250514 | 142 | 3 | 2.1% | 8,200 | 4.2s |
| gpt-4o | 28 | 5 | 17.9% | 12,400 | 6.8s |
**Top failing tools**: edit (4 errors), bash (3 errors), read (1 error)
Want a deep analysis with wiki cross-reference and per-model recommendations?
Troubleshooting
| Problem | Likely cause | Fix |
|---|
/posthog-mcp-status fails | MCP bridge not configured | export POSTHOG_PERSONAL_API_KEY=phx_... then restart pi |
| Queries return empty | No tracing data captured | Set POSTHOG_API_KEY and generate some agent sessions |
execute-sql tool not found | MCP tool list not loaded | Run /posthog-mcp-reload |
| Query syntax errors | PostHog uses ClickHouse SQL | Wrap column names in double quotes if they contain special chars |
| Only one model in results | Only one model used in the date range | Extend date range or note that comparison isn't possible |
Token Discipline
- Step 1 (mcp-status): ~200 tokens
- Step 2 (queries): response depends on result size. Use spill-to-file (enabled by default at 12K chars).
- Step 3-4 (wiki cross-ref): Read only relevant wiki pages. ~300 tokens each.
- Step 5-6 (recommend + file): ~2,000 tokens for analysis page
Total: ~3,000-5,000 tokens for a typical analysis. If query results are large, PostHog MCP spills them to /tmp/posthog-mcp/ automatically.
