| name | creating-online-evaluations |
| description | Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure mode you've identified. Use when the user wants an evaluation that automatically scores new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn this failure into an eval". Covers choosing the target and eval type (hog / llm_judge / sentiment), configuring a provider, model, and usable provider key for an llm_judge eval, scoping which generations trigger it via conditions (property filters + rollout sampling), creating it disabled, verifying scope, and enabling. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.
|
Creating online evaluations
An online evaluation automatically scores either each matching $ai_generation or the whole trace
containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic,
not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once that
failure mode is identified and turns it into a scoped, continuously-running eval.
First, know what you're evaluating. Finding and ranking the failure modes worth catching is a
separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague
about it and don't refer to a specific failure mode, run exploring-ai-failures to scope a use case,
find failing traces, and produce a ranked list of failure modes.
For the mechanics of writing and iterating an evaluator (Hog source vs LLM-judge prompt, dry-running,
debugging a live eval), defer to exploring-llm-evaluations.
Tools
| Tool | Purpose |
|---|
posthog:llma-evaluation-config-get | Check the active provider key used by unpinned judges |
posthog:llma-provider-key-list | Find a usable (ok state) provider key to pin |
posthog:llma-evaluation-judge-models | List valid provider+model combos |
posthog:llma-evaluation-test-hog | Dry-run Hog source against recent generations before creating |
posthog:llma-evaluation-create | Create the evaluation (always enabled: false first) |
posthog:llma-evaluation-run | Spot-run a draft eval against one generation |
posthog:llma-evaluation-update | Iterate config, then flip enabled: true |
posthog:execute-sql | Verify a condition matches the events and volume you expect |
posthog:generate-app-url | Build a region- and project-qualified deep link to the eval |
The full create payload (every field, the config schemas, the exact conditions shape) is in
references/evaluation-payload.md.
Phase 1 — Pick the failure mode to evaluate
Start from a real, observed failure, not a metric you picked in advance. If you don't already have one,
run exploring-ai-failures to scope a use case, find failing traces, and produce a ranked list of failure
modes — then come back. With that list in hand, talk with the user to choose what to turn into an eval:
- Most frequent, most painful first. A handful of modes usually cover the majority of failures.
- Pair obvious fixes with the eval, don't skip it. If a prompt tweak would likely fix the failure, set
up the eval anyway and suggest the fix alongside it — a rising pass rate is how you confirm the fix landed.
- One mode per eval. Three failure modes is three evals, not one prompt trying to catch everything.
You should end with a single, crisp, checkable criterion — "the reply must stay on the user's topic", "the
tool call must include an order_id". Then move to Phase 2.
Phase 2 — Build the online eval
2.1 — Choose the eval type
| Use… | When the criterion is… |
|---|
hog | Structural / rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, no provider key needed. |
llm_judge | Subjective / fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider, model, and usable provider key. |
sentiment | You want sentiment labels on user messages, not a pass/fail (unless very specifically asked for, usually not relevant to this skill). |
Reach for hog first, escalate to llm_judge if there is no deterministic way to check for what we want to check.
2.2 — Choose the target
| Target | Behavior |
|---|
generation | Runs once for each matching $ai_generation, immediately after ingestion. This is the default. |
trace | Runs once for the whole trace after the first matching generation and a configurable wait for the trace to finish. |
For a trace target, send "target": "trace" plus a settle config that controls when the trace is
evaluated, discriminated on strategy:
{ "strategy": "fixed_window", "window_seconds": 1800 } — evaluate a fixed wait after the first
matching generation. Between 10 seconds and 2 hours, defaults to 30 minutes. A target_config
without a strategy key means this.{ "strategy": "inactivity", "quiet_period_seconds": 300, "max_age_seconds": 7200 } — evaluate once
the trace has had no new activity for the quiet period (10 seconds to 30 minutes,
defaults to 5 minutes). max_age_seconds caps the total wait from the first matching generation
(1 minute to 2 hours, defaults to 2 hours, must be at least the quiet period).
Conditions still match the generation that triggers the run; the evaluator itself receives
the complete trace. Sentiment evaluations support only the generation target.
New Hog source should use the globals shared by both targets:
| Global | Meaning |
|---|
evaluation_events | One generation event for a generation target, or every captured event for a trace target. |
target | The target's type, id, total_cost_usd, and total_latency_seconds. |
item.input_text / item.output_text | Best-effort readable projections; use these for length, keyword, and regex checks. |
item.input / item.output | Original serialized values; use these when the evaluator needs to parse the captured JSON itself. |
Generation evaluations still expose top-level input, output, properties, and event. Trace evaluations
still expose their original events and trace globals. Those globals are kept for compatibility with saved
evaluators. Do not use target-specific globals in new source that needs to work for both targets. The text
projections recognize common provider payloads but are not authoritative; use item.input / item.output when
exact structure matters.
2.3 — Configure the LLM judge
An llm_judge evaluation requires a valid provider and model. It also needs a usable provider key
when it runs. provider_key_id controls whether the evaluation pins one specific key:
- Set
provider_key_id to the UUID of an ok-state key for the same provider to pin it.
- Set
provider_key_id to null to use the team's active provider key. The active key must be in the
ok state and use the same provider as model_configuration.provider.
Hog and sentiment evaluations skip this step.
posthog:llma-evaluation-config-get
posthog:llma-provider-key-list
posthog:llma-evaluation-judge-models
Confirm the provider and model with llma-evaluation-judge-models. Prefer pinning the chosen key so a later
team-wide active-key change does not change how the evaluation runs. Leave provider_key_id as null only
after llma-evaluation-config-get confirms the active key is usable and its provider matches.
If there is no usable key, you may still create a disabled draft for the user to review. Do not spot-run or
enable it. Ask the user to add or validate a key in the UI before continuing.
2.4 — Create it disabled
Create with enabled: false so nothing fires until the scope is verified. Minimal hog example:
posthog:llma-evaluation-create
{
"name": "Output is not empty",
"description": "Fails when a generation has no readable output",
"evaluation_type": "hog",
"evaluation_config": { "source": "let count := 0\nfor (let i, item in evaluation_events) {\n if (item.event == '$ai_generation') {\n count := count + 1\n if (length(trim(item.output_text)) == 0) { return false }\n }\n}\nreturn count > 0" },
"output_type": "boolean",
"output_config": { "allows_na": false },
"target": "generation",
"target_config": {},
"conditions": [
{ "id": "default", "rollout_percentage"
For llm_judge, swap evaluation_config to { "prompt": "…" } and add
"model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": "<uuid of an ok-state key from llma-provider-key-list>" }.
Use null only when the active team key is ok and uses the same provider. Full field reference:
references/evaluation-payload.md.
2.5 — Verify the scope before enabling
conditions is where online evals go wrong: too broad and you evaluate (and bill) a firehose; too narrow
and it never fires. Confirm the filter matches the events you expect, and roughly how many per day:
posthog:execute-sql
SELECT count() AS matched, count() / 7 AS per_day
FROM events
WHERE event = '$ai_generation'
AND properties.$ai_model ILIKE '%gpt%'
AND timestamp >= now() - INTERVAL 7 DAY
For generation targets, count() is the run volume. For trace targets, count distinct non-empty
$ai_trace_id values because matching generations from the same trace schedule only one run.
If volume is high, set rollout_percentage below 100 to sample. Spot-check the evaluator with
llma-evaluation-test-hog (hog) or llma-evaluation-run against one generation (llm_judge).
Both tools currently use generation samples; for a trace target they can check shared source or prompt behavior,
but they do not reproduce the complete settled trace. Review the first live trace results before increasing rollout.
Watch out: some orgs reuse a single $ai_trace_id across 100k+ events. Scoping by trace-ID prefix
can match far more than expected — verify volume with the SQL above before enabling.
2.6 — Enable, then close the loop
posthog:llma-evaluation-update
{ "evaluationId": "<uuid>", "enabled": true }
It now runs on every new matching generation, or once per matching trace for a trace target. This isn't
one-and-done: the user should be aware that they need to keep an eye on results and iterate if the outcome
is not the expected one. To wire results into a Slack feed, see feature-usage-feed.
Scoping with conditions
conditions is a list of condition sets — OR between sets, AND within a set's properties. Each
set is { id, rollout_percentage, properties[] }. There is no time window inside conditions; sampling is
only rollout_percentage (0–100). Property filters use the standard PostHog shape
(key, type, operator, value). For trace targets, these filters still select the generation that
triggers the eventual whole-trace evaluation.
"conditions": [
{ "id": "openai", "rollout_percentage": 100, "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "openai"}] },
{ "id": "anthropic", "rollout_percentage": 25, "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact",
Constructing UI links
Build links with posthog:generate-app-url — never hand-write the host or the /project/<id>/ prefix.
The url must be a canonical catalog template; pass concrete ids via params, never inline them into the path.
- Evaluations list:
generate-app-url {url: "/ai-evals/evaluations"}
- Single evaluation:
generate-app-url {url: "/ai-evals/evaluations/{id}", params: {id: "<evaluation_id>"}}
These resolve to the correct region host and project prefix (e.g.
https://us.posthog.com/project/<id>/ai-evals/evaluations/<evaluation_id>). Surface the link after
creating so the user can review and toggle it in the UI.
Tips
- Evals come from real failures, not generic metrics. Start from a failure found in this product's
traffic (via
exploring-ai-failures), not from "let's measure hallucination". A metric nobody traced
back to a real bad output is noise.
- One eval, one failure mode. Different failure modes need different evals; don't make one eval try to
catch everything.
- Suggest changes along with the eval if possible. If it's clear a prompt change would fix the issue, for
instance, set up the eval but also suggest to the user they change the prompt: they should soon see the eval
go from low pass rate to a higher pass rate.
hog first. No provider key, no AI approval, deterministic. Reach for llm_judge only when the
criterion genuinely can't be coded.- Always create disabled, verify scope, then enable. An eval firing on the wrong events is worse than
none — noise, and (for llm_judge) cost.
- Configure llm_judge credentials before running. A judge needs a valid provider and model plus a usable
provider key.
provider_key_id may be null only when the matching active team key can be used.
bytecode is server-written for hog evals — never pass it; send only evaluation_config.source.- For cluster-scoped evals, identify the cluster with
exploring-llm-clusters, then translate its event
filter into conditions.
