creating-online-evaluations

# Creating online evaluations An **online evaluation** automatically scores either each matching `$ai_generation` or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once those failure modes are identified and turns them into scoped, continuously-running evals. **One eval per failure mode, and as many evals as the data justifies.** How many to create is a judgment call you make from what the traces actually showed — sometimes one, often three or four. Never assume the answer is one, and never bundle several modes into one evaluator. **Propose before you create.** Bring the user a short list of candidate evals and let them pick which ones they want (Phase 1.1). Creating evals they didn't ask for costs them money and noise. **First, know what you're evaluating.** Finding and ranking the failure modes worth catching is a separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague about it and don't refer to a specific failure mode, run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure modes. For the mechanics of _writing and iterating_ an evaluator (Hog source vs LLM-judge prompt, dry-running, debugging a live eval), defer to `exploring-llm-evaluations`. ## Tools | Tool | Purpose | | ------------------------------------------- | ------------------------------------------------------------- | | `posthog:llma-evaluation-config-get` | Check the active provider key used by unpinned judges | | `posthog:llma-provider-key-list` | Find a usable (`ok` state) provider key to pin | | `posthog:llma-evaluation-judge-models` | List valid provider+model combos | | `posthog:llma-evaluation-directory-list` | List directories available for organizing the evaluation | | `posthog:llma-evaluation-directory-create` | Create a directory when the user asks for a new one | | `posthog:llma-evaluation-test-hog` | Dry-run Hog source against recent generations before creating | | `posthog:llma-evaluation-create` | Create the evaluation (always `enabled: false` first) | | `posthog:llma-evaluation-run` | Spot-run a draft eval against one generation | | `posthog:llma-evaluation-update` | Iterate config, then flip `enabled: true` | | `posthog:llma-evaluation-backfill-estimate` | Count what a backfill over past data would evaluate | | `posthog:llma-evaluation-backfill-create` | Evaluate past data, only after the user confirms the estimate | | `posthog:llma-evaluation-backfill-get` | Track a backfill's progress and coverage | | `posthog:llma-evaluation-backfill-list` | Find an evaluation's backfills, including one already running | | `posthog:llma-evaluation-backfill-cancel` | Stop a running backfill when the user asks | | `posthog:execute-sql` | Verify a condition matches the events and volume you expect | | `posthog:generate-app-url` | Build a region- and project-qualified deep link to the eval | The full create payload (every field, the config schemas, the exact `conditions` shape) is in [references/evaluation-payload.md](references/evaluation-payload.md). ## Phase 1 — Decide what to propose, then let the user choose Start from real, observed failures, not metrics you picked in advance. If you don't already have them, run `exploring-ai-failures` to scope a use case, find failing traces, and produce a ranked list of failure modes — then come back. ### 1.1 — Turn the failure modes into a candidate set **Let the data decide how many.** One failure mode is one eval, so a ranked list of four distinct modes is a candidate set of four evals. Don't collapse them into one evaluator that tries to catch everything, and don't stop at the top mode when the traces clearly showed more worth watching. Keep a candidate when: - **It hurts.** Frequent or painful. A handful of modes usually covers the majority of failures. - **It's checkable.** It reduces to one crisp criterion — "the reply must stay on the user's topic", "the tool call must include an `order_id`". If you can't state it in a line, it isn't ready to propose. - **It's distinct.** Two candidates that would fail on the same generations are one eval. Rank by how much they hurt and propose roughly the top five; mention in a line that you set weaker ones aside rather than silently dropping them. ### 1.2 — Propose the set in plain language **Never create evals the user hasn't picked.** Lay out the candidates and ask which ones they want. Assume they haven't read the traces with you and don't know the eval vocabulary. Keep each one to a line or two — a wall of text per eval means they can't compare them — but make it obvious what it watches and what they'll see when it fails. No Hog snippets, property filters, or `hog`/`llm_judge` internals here. Number them so they can reply "1 and 3": > Found 3 failure patterns worth watching. Which should I set up? > > **1. Replies drift off topic** — checks the answer addresses what the user actually asked. > Catches support replies that confidently answer a different question. Seen ~40×/day. > > **2. Order lookups missing an order ID** — checks every `lookup_order` call includes an `order_id`. > Catches the silent tool failures that make the agent invent an order status. Seen ~15×/day. > > **3. Refuses questions it can answer** — checks the reply isn't a needless "I can't help with that". > Catches users being turned away from things the docs cover. Costs an LLM call per check. Per eval: what it checks, what bad thing it surfaces, and rough volume when you know it (Phase 2.5 verifies it properly). Flag the ones needing an LLM judge, since those cost per run — that's the only mechanic worth exposing up front. Keep the ranking implicit in the order; skip scoring tables. Recommend a starting point if they seem unsure (usually the top one or two), and treat "all of them" or "go ahead" as accepting the whole set. If a prompt tweak would likely fix a mode, suggest the fix alongside the eval rather than in place of it — a rising pass rate is how they confirm the fix landed. ### 1.3 — When nothing surfaced, propose sentiment If the traces were read and no failure mode is worth an eval, don't invent a failure and don't come back empty-handed. Say what you looked at, then propose a `sentiment` eval as the floor: > No clear failure pattern in the last 7 days. Worth starting with: > > **1. User frustration** — labels each user message positive, neutral, or negative. > Shows which conversations are going badly, which is usually where the real failures hide. No judge cost. It needs no provider key, it's cheap, and it gives them a signal to come back to once there's enough traffic to spot patterns. **No generations means no eval of any kind.** Sentiment only scores matching `$ai_generation` events, so if the project has none, every eval you could create — sentiment included — would sit there never firing. Don't propose one. Point them at `instrument-llm-analytics` to get AI observability capturing generations first, and come back to this skill once there's traffic to read. ## Phase 2 — Build each accepted eval Run 2.1 through 2.5 once per eval the user picked, so each one lands as a verified draft. **Leave every one of them disabled until the whole set is verified** — 2.6 is a single pass over the finished set at the end, not the last step of each loop. Enabling eval 1 while eval 3 is still being written puts a partially live set into production, which is noise and (for a judge) cost the user didn't agree to yet. ### 2.1 — Choose the eval type | Use… | When the criterion is… | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `hog` | Structural / rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, **no provider key needed.** | | `llm_judge` | Subjective / fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider and model. A usable provider key is only needed to enable it, not to create a draft. | | `sentiment` | You want sentiment labels on user messages, not a pass/fail (unless very specifically asked for, usually not relevant to this skill). | Reach for `hog` first, escalate to `llm_judge` if there is no deterministic way to check for what we want to check. ### 2.2 — Choose the target | Target | Behavior | | ------------ | ------------------------------------------------------------------------------------------------------------------ | | `generation` | Runs once for each matching `$ai_generation`, immediately after ingestion. This is the default. | | `trace` | Runs once for the whole trace after the first matching generation and a configurable wait for the trace to finish. | | `session` | Runs once for the whole `$ai_session_id` session, after the session settles. | For a trace target, send `"target": "trace"` plus a settle config that controls when the trace is evaluated, discriminated on `strategy`: - `{ "strategy": "fixed_window", "window_seconds": 1800 }` — evaluate a fixed wait after the first matching generation. Between 10 seconds and 2 hours, defaults to 30 minutes. A `target_config` without a `strategy` key means this. - `{ "strategy": "inactivity", "quiet_period_seconds": 300, "max_age_seconds": 7200 }` — evaluate once the trace has had no new activity for the quiet period (10 seconds to 30 minutes, defaults to 5 minutes). `max_age_seconds` caps the total wait from the first matching generation (1 minute to 2 hours, defaults to 2 hours, must be at least the quiet period). A `session` target takes the same settle config with session-sized bounds, and defaults to `inactivity` rather than `fixed_window`: - `{ "strategy": "inactivity", "quiet_period_seconds": 3600, "max_age_seconds": 86400 }` — evaluate once the session has had no new activity for the quiet period (10 seconds to 24 hours, defaults to 1 hour). `max_age_seconds` caps the total wait from the first matching generation (1 minute to 7 days, defaults to 24 hours, must be at least the quiet period). - `{ "strategy": "fixed_window", "window_seconds": 1800 }` — evaluate a fixed wait after the first matching generation (10 seconds to 7 days). A session evaluation only fires for events that carry `$ai_session_id`. Producers either set it on every generation or on none, so an SDK that does not set it will never trigger a session evaluation. `$ai_session_id` is not `$session_id`: the second is PostHog's product-analytics session and is unrelated. A session evaluation can also come back skipped rather than graded. The emitted `$ai_evaluation` event then carries `$ai_evaluation_skipped: true` and an `$ai_evaluation_skip_reason`, and its `$ai_evaluation_result` is `false` when the evaluation disallows N/A, so any analysis of pass rates has to exclude skipped runs rather than count them as failures. Sessions are skipped when they hold more than 2500 events (usually a session id shared across conversations), when nothing was found in the evaluation window, and, for an LLM judge, when the transcript is too long to send in full. A session is evaluated at most once per evaluation, for as long as the completed run stays inside Temporal's retention window. A session that resumes long after being evaluated may be evaluated again, so pick a quiet period long enough that the session is really finished. A longer quiet period costs only latency. Conditions still match the generation that triggers the run; the evaluator itself receives the complete trace or session. Sentiment evaluations support only the generation target. New Hog source should use the globals shared by all targets: | Global | Meaning | | -------------------------------------- | ---------------------------------------------------------------------------------------------------- | | `evaluation_events` | One generation event for a generation target, or every captured event for a trace or session target. | | `target` | The target's `type`, `id`, `total_cost_usd`, and `total_latency_seconds`. | | `item.input_text` / `item.output_text` | Best-effort readable projections; use these for length, keyword, and regex checks. | | `item.input` / `item.output` | Original serialized values; use these when the evaluator needs to parse the captured JSON itself. | For a session target, `target.id` is the session id, and `target.total_cost_usd` / `target.total_latency_seconds` are summed across the session's traces. `total_latency_seconds` is time spent on AI work, not session wall-clock; the two can differ by orders of magnitude. Session wall-clock is derivable from `evaluation_events` timestamps. Generation evaluations still expose top-level `input`, `output`, `properties`, and `event`. Trace evaluations still expose their original `events` and `trace` globals. Those globals are kept for compatibility with saved evaluators. Session evaluations do not carry them: session Hog source only receives `target` and `evaluation_events`. Do not use target-specific globals in new source that needs to work across targets. The text projections recognize common provider payloads but are not authoritative; use `item.input` / `item.output` when exact structure matters. ### 2.3 — Configure the LLM judge An `llm_judge` evaluation requires only a valid `provider` and `model` to create as a draft. A usable provider key is needed to enable it, not to save it, so no `ok` key is a reason to keep the eval disabled, not a reason to stop. `provider_key_id` controls whether the evaluation pins one specific key: - Set `provider_key_id` to the UUID of an `ok`-state key for the same provider to pin it. - Set `provider_key_id` to `null` to use the team's active provider key. This is always a valid saved value. To enable the eval, the active key must be in the `ok` state and use the same provider as `model_configuration.provider`. Hog and sentiment evaluations skip this step. ```json posthog:llma-evaluation-config-get // check active_provider_key for an unpinned judge posthog:llma-provider-key-list // find an ok-state key to pin posthog:llma-evaluation-judge-models // {} → every provider and its models; { "provider": "openai" } narrows it ``` Confirm the provider and model with `llma-evaluation-judge-models`. Call it with no arguments to see the whole catalog at once. Providers PostHog funds no models for come back empty unless you pass `key_id` for one of the team's keys; the response's `providers` list flags which ones those are. Prefer pinning the chosen key so a later team-wide active-key change does not change how the evaluation runs. Leave `provider_key_id` as `null` to run on the team's active key; before enabling, confirm with `llma-evaluation-config-get` that the active key is usable and its provider matches. No usable key does not block creation. Create the disabled draft with a valid `provider` and `model` for the user to review. Do not spot-run or enable it, and ask the user to add or validate a key in the UI so it can be enabled later. ### 2.4 — Create it disabled Create with `enabled: false` so nothing fires until the scope is verified. Minimal `hog` example: Evaluations may be created at the top level or in one directory. If the user names a directory, call `posthog:llma-evaluation-directory-list` and pass its UUID as `directory_id`. Create a directory only when the user asks for one. Omit `directory_id` or pass `null` for the top level. Directories cannot be nested. ```json posthog:llma-evaluation-create { "name": "Output is not empty", "description": "Fails when a generation has no readable output", "evaluation_type": "hog", "evaluation_config": { "source": "let count := 0\nfor (let i, item in evaluation_events) {\n if (item.event == '$ai_generation') {\n count := count + 1\n if (length(trim(item.output_text)) == 0) { return false }\n }\n}\nreturn count > 0" }, "output_type": "boolean",