aio11y

name	aio11y
description	Use when the user wants to list or search AI Observability conversations, inspect generations, manage evaluators (create, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like "list conversations", "search generations", "what did the agent do", "debug LLM conversation", "create evaluator", "set up evaluation rule", "test evaluator", "check scores", "evaluate generation quality", or "set up online evaluation".
allowed-tools	["gcx","Bash","Read","Write","Edit"]

AI Observability

AI Observability is Grafana's AI observability platform. It records what LLM-powered applications do in production and scores the quality of their output.

Applications send generations (individual LLM API calls — request, response, model, tokens, tool calls) to AI Observability. Generations belonging to the same user session are grouped into a conversation.

Evaluators are scoring functions (LLM judge, regex, heuristic, JSON schema, etc.) that assess generation quality. Rules bind evaluators to production traffic, they select which generations to evaluate (e.g. only user-visible turns), filter by agent/model, and control sampling rate. When a rule matches a generation, AI Observability runs the bound evaluators and writes scores.

All commands live under gcx aio11y. Use gcx aio11y <subcommand> --help for flags and usage.

Command Groups

Group	Purpose
conversations	List, get, search conversations
generations	Get a single generation