exploring-llm-evaluations

# Exploring AI observability evaluations PostHog evaluations score `$ai_generation` events. Each evaluation is one of three types: - **`hog`** — deterministic Hog code that returns a boolean, numeric score, or categories (and optionally N/A). Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code. - **`llm_judge`** — an LLM scores generations against a prompt you write. Best for subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level. - **`sentiment`** — classifies sentiment from user messages on each matching generation. Returns a sentiment label and score, not a pass/fail verdict. Results from all types land in ClickHouse as `$ai_evaluation` events. Boolean evaluations (`llm_judge` and `hog`) set `$ai_evaluation_result`; sentiment evaluations set `$ai_sentiment_*` properties instead. Both `hog` and `llm_judge` also support `output_type: "numeric"`. Numeric runs store their raw score in `$ai_evaluation_numeric_result`, with optional `$ai_evaluation_numeric_result_min` and `$ai_evaluation_numeric_result_max`. They never set `$ai_evaluation_result`. Use `output_config.passing_rule` to interpret scores: `gte` means at least the threshold and `lte` means at most. Changing the rule reinterprets historical scores. Saved reports retain the rule and metrics used when generated. Without a passing rule, inspect score means and distributions; pass rates and reports are unavailable. Both runtimes also support `output_type: "categorical"` when categorical evaluations are enabled for the project. Read `output_config.options` for the configured `{key, label}` pairs and `selection_mode` for `single` or `multiple`. Results store keys as a list in `$ai_evaluation_categorical_result`; single selection requires exactly one key, while multiple selection accepts `[]`. Hog can also return a single key string in single mode. Use `output_config.passing_rule.categories` to grade results: a nonempty result passes only when every returned key is a passing category. A nonempty rule fails `[]`; an empty rule passes only `[]` and is allowed only in multiple mode. Without a rule, inspect category frequencies instead of pass rates; new reports are unavailable. Rule edits reclassify historical results, while saved reports retain their rule snapshot. This skill covers the full lifecycle: list/inspect/manage evaluation configs, run them on specific generations, query individual results, and configure evaluation reports that summarize recent runs on a schedule. ## Tools | Tool | Purpose | | ----------------------------------------- | -------------------------------------------------------------- | | `posthog:llma-evaluation-list` | List/search evaluation configs (filter by name, enabled flag) | | `posthog:llma-evaluation-get` | Get a single evaluation config by UUID | | `posthog:llma-evaluation-create` | Create a new `llm_judge`, `hog`, or `sentiment` evaluation | | `posthog:llma-evaluation-update` | Update an existing evaluation (name, prompt, enabled, …) | | `posthog:llma-evaluation-delete` | Soft-delete an evaluation | | `posthog:llma-evaluation-run` | Run an evaluation against a specific `$ai_generation` event | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations (no save) | | `posthog:llma-evaluation-report-list` | List the report configs attached to an evaluation | | `posthog:llma-evaluation-report-create` | Schedule an AI report on an evaluation (email or Slack) | | `posthog:llma-evaluation-report-run-list` | Past report runs, including the report content that was sent | | `posthog:execute-sql` | Ad-hoc HogQL over `$ai_evaluation` events | | `posthog:query-llm-trace` | Drill into the underlying generation that an evaluation scored | All `llma-evaluation-*` tools are defined in `products/ai_observability/mcp/tools.yaml`. ## Event schema Every run of an evaluation emits an `$ai_evaluation` event. Key properties: | Property | Meaning | | ----------------------------------- | -------------------------------------------------------------------------------- | | `$ai_evaluation_id` | UUID of the evaluation config | | `$ai_evaluation_name` | Human-readable name | | `$ai_target_event_id` | UUID of the `$ai_generation` event being scored | | `$ai_trace_id` | Parent trace ID (for jumping to the trace UI) | | `$ai_evaluation_result_type` | Result kind: `boolean`, `numeric`, `categorical`, or `sentiment` | | `$ai_evaluation_numeric_result` | Raw numeric score. Use the passing rule to map it to pass or fail | | `$ai_evaluation_categorical_result` | List of category keys. Use the passing rule to map it to pass or fail | | `$ai_evaluation_result` | Raw boolean result. Use the evaluation's output config to map it to pass or fail | | `$ai_evaluation_reasoning` | Free-text explanation (set by the LLM judge or Hog code) | | `$ai_evaluation_applicable` | `false` when the evaluator decided the generation is N/A | | `$ai_evaluation_skipped` | `true` when no result was graded because the run could not be evaluated | | `$ai_sentiment_label` | For sentiment evaluations: `positive`, `neutral`, or `negative` | | `$ai_sentiment_score` | Confidence score for the winning sentiment label | Check `$ai_evaluation_skipped` first: skipped runs are excluded from pass rates and are distinct from N/A. Otherwise, `$ai_evaluation_applicable = false` means N/A regardless of the result value. For evaluations that don't support N/A, this property may be `null` — treat null as "applicable". For boolean evaluations, `output_config.true_is_failure: false` maps `true` to pass and `false` to fail. Set it to `true` for detector-style evaluations where `true` means the evaluator found a problem. ## Workflow: investigate why an evaluation is failing Works the same way for boolean `llm_judge` and `hog` evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source). Sentiment evaluations should be inspected by sentiment label and score rather than pass/fail filters. The SQL pass/fail examples below use boolean results. For numeric and categorical evaluations, read their result property and current passing rule instead of `$ai_evaluation_result`. ### Step 1 — Find the evaluation ```json posthog:llma-evaluation-list { "search": "hallucination", "enabled": true } ``` Look at the returned `id`, `name`, `evaluation_type`, and either: - `evaluation_config.prompt` for an `llm_judge` - `evaluation_config.source` for a `hog` evaluator The Hog source is the ground truth for why a hog evaluator passes or fails — read it before assuming the failure is in the generation. ### Step 2 — Break down pass, fail, and N/A ```sql posthog:execute-sql SELECT countIf(properties.$ai_evaluation_applicable = false) AS na_count, countIf( (properties.$ai_evaluation_applicable IS NULL OR properties.$ai_evaluation_applicable != false) AND properties.$ai_evaluation_result = true ) AS pass_count, countIf( (properties.$ai_evaluation_applicable IS NULL OR properties.$ai_evaluation_applicable != false) AND properties.$ai_evaluation_result = false ) AS fail_count FROM events WHERE event = '$ai_evaluation' AND properties.$ai_evaluation_id = '<evaluation_uuid>' AND timestamp >= now() - INTERVAL 7 DAY ``` If the evaluation already has report configs, `llma-evaluation-report-list` and `llma-evaluation-report-run-list` give you the AI-written reports from earlier periods, which is a fast way to see how the picture has moved. ### Step 3 — Read the failing runs The reasoning text is where the pattern shows up. Pull the recent fails and read them: ```sql posthog:execute-sql SELECT properties.$ai_target_event_id AS generation_id, properties.$ai_trace_id AS trace_id, properties.$ai_evaluation_reasoning AS reasoning, timestamp FROM events WHERE event = '$ai_evaluation' AND properties.$ai_evaluation_id = '<evaluation_uuid>' AND properties.$ai_evaluation_result = false AND ( properties.$ai_evaluation_applicable IS NULL OR properties.$ai_evaluation_applicable != false ) AND timestamp >= now() - INTERVAL 7 DAY ORDER BY timestamp DESC LIMIT 25 ``` The N/A guard (`IS NULL OR != false`) is important — it matches the same logic the backend uses to bucket runs. ### Step 4 — Drill into example failing runs Take the most representative rows from Step 3 and pull the underlying trace: ```json posthog:query-llm-trace { "traceId": "<trace_id>", "dateRange": {"date_from": "-30d"} } ``` (If you only have a generation ID, query for it via `execute-sql` first to find the parent trace ID.) ## Workflow: run an evaluation against a specific generation Use this when the user pastes a trace/generation URL and asks "what would evaluation X say about this?". ```json posthog:llma-evaluation-run { "evaluationId": "<eval_uuid>", "target_event_id": "<generation_event_uuid>", "timestamp": "2026-04-01T19:39:20Z", "event": "$ai_generation" } ``` The `timestamp` is required for an efficient ClickHouse lookup of the target event. Pass `distinct_id` if you have it — it speeds up the lookup further. ## Workflow: build and test a new evaluator ### Hog evaluator (deterministic, code-based) Reach for this first when the criterion is rule-based — it's cheaper, faster, and reproducible. Prototype with `llma-evaluation-test-hog` (no save): ```json posthog:llma-evaluation-test-hog { "source": "return event.properties.$ai_output_choices[1].content contains 'sorry';", "sample_count": 5, "allows_na": false } ``` The handler returns raw boolean or numeric results for the sampled units. For numeric previews, send `output_type: "numeric"` and the saved `output_config`, including `allows_na: true` if null is allowed. Iterate on the source until it behaves as expected, then promote it via `llma-evaluation-create`: ```json posthog:llma-evaluation-create { "name": "Output is valid JSON", "description": "Fails when the assistant message can't be parsed as JSON", "evaluation_type": "hog", "evaluation_config": { "source": "let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }" }, "output_type": "boolean", "enabled": true } ``` Hog evaluators have full access to the event and its properties — common patterns include schema validation, length/token limits, regex matches, and tool-call shape checks. Because they're deterministic, results are reproducible across reruns and trivially diff-able. ### LLM-judge evaluator (subjective, prompt-based) Use this when the criterion is fuzzy and a code rule would be brittle (tone, factuality, helpfulness, on-topic-ness). There's no equivalent of `llma-evaluation-test-hog` for LLM judges — the typical loop is to create the evaluator with `enabled: false`, run it manually against a handful of representative generations via `llma-evaluation-run`, inspect the results, refine the prompt with `llma-evaluation-update`, and then flip `enabled: true` when you're satisfied: ```json posthog:llma-evaluation-create { "name": "Response stays on-topic", "description": "LLM judge — fails if the assistant changes topic from the user's question", "evaluation_type": "llm_judge", "evaluation_config": { "prompt": "You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N/A if the user did not actually ask a question." }, "output_type": "boolean", "output_config": { "allows_na": true }, "model_configuration": { "provider": "openai", "model": "gpt-5-mini" }, "enabled": false } ``` Then dry-run against a known-good and a known-bad generation: ```json posthog:llma-evaluation-run { "evaluationId": "<new_eval_uuid>", "target_event_id": "<generation_uuid>", "timestamp": "2026-04-01T19:39:20Z" } ``` LLM judges require organisation AI data processing approval. Hog evaluators do not. ## Workflow: manage the evaluation lifecycle | Action | Tool | | -------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Add a Hog evaluator | `llma-evaluation-create` with `evaluation_type: "hog"` and `evaluation_config.source` | | Add an LLM-judge evaluator | `llma-evaluation-create` with `evaluation_type: "llm_judge"`, `evaluation_config.prompt`, and a `model_configuration` | | Tweak the source or prompt | `llma-evaluation-update` (edits `evaluation_config.source` for Hog, `evaluation_config.prompt` for LLM judge) |