arize-evaluator

# Arize Evaluator Skill > **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`. This skill covers designing, creating, and running **LLM-as-judge evaluators** on Arize. An evaluator defines the judge; a **task** is how you run it against real data. --- ## Prerequisites Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront. If an `ax` command fails, troubleshoot based on the error: - `command not found` or version error → see references/ax-setup.md - `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys - Space unknown → run `ax spaces list` to pick by name, or ask the user - LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the **arize-ai-provider-integration** skill - **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user. - **CRITICAL — Never fabricate evaluation results:** If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with `ax ai-integrations list`, (4) contact support at https://arize.com/support --- ## Concepts ### What is an Evaluator? An **evaluator** is an LLM-as-judge definition. It contains: | Field | Description | |-------|-------------| | **Template** | The judge prompt. Uses `{variable}` placeholders (e.g. `{input}`, `{output}`, `{context}`) that get filled in at run time via a task's column mappings. | | **Classification choices** | The set of allowed output labels (e.g. `factual` / `hallucinated`). Binary is the default and most common. Each choice can optionally carry a numeric score. | | **AI Integration** | Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. | | **Model** | The specific judge model (e.g. `gpt-4o`, `claude-sonnet-4-5`). | | **Invocation params** | Optional JSON of model settings like `{"temperature": 0}`. Low temperature is recommended for reproducibility. | | **Optimization direction** | Whether higher scores are better (`maximize`) or worse (`minimize`). Sets how the UI renders trends. | | **Data granularity** | Whether the evaluator runs at the **span**, **trace**, or **session** level. Most evaluators run at the span level. | Evaluators are **versioned** — every prompt or model change creates a new immutable version. The most recent version is active. ### What is a Task? A **task** is how you run one or more evaluators against real data. Tasks are attached to a **project** (live traces/spans) or a **dataset** (experiment runs). A task contains: | Field | Description | |-------|-------------| | **Evaluators** | List of evaluators to run. You can run multiple in one task. | | **Column mappings** | Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. `"input" → "attributes.input.value"`). This is what makes evaluators portable across projects and experiments. | | **Query filter** | SQL-style expression to select which spans/runs to evaluate (e.g. `"span_kind = 'LLM'"`). Optional but important for precision. | | **Continuous** | For project tasks: whether to automatically score new spans as they arrive. | | **Sampling rate** | For continuous project tasks: fraction of new spans to evaluate (0–1). | --- ## Data Granularity The `--data-granularity` flag controls what unit of data the evaluator scores. It defaults to `span` and only applies to **project tasks** (not dataset/experiment tasks — those evaluate experiment runs directly). | Level | What it evaluates | Use for | Result column prefix | |-------|-------------------|---------|---------------------| | `span` (default) | Individual spans | Q&A correctness, hallucination, relevance | `eval.{name}.label` / `.score` / `.explanation` | | `trace` | All spans in a trace, grouped by `context.trace_id` | Agent trajectory, task correctness — anything that needs the full call chain | `trace_eval.{name}.label` / `.score` / `.explanation` | | `session` | All traces in a session, grouped by `attributes.session.id` and ordered by start time | Multi-turn coherence, overall tone, conversation quality | `session_eval.{name}.label` / `.score` / `.explanation` | ### How trace and session aggregation works For **trace** granularity, spans sharing the same `context.trace_id` are grouped together. Column values used by the evaluator template are comma-joined into a single string (each value truncated to 100K characters) before being passed to the judge model. For **session** granularity, the same trace-level grouping happens first, then traces are ordered by `start_time` and grouped by `attributes.session.id`. Session-level values are capped at 100K characters total. ### The `{conversation}` template variable At session granularity, `{conversation}` is a special template variable that renders as a JSON array of `{input, output}` turns across all traces in the session, built from `attributes.input.value` / `attributes.llm.input_messages` (input side) and `attributes.output.value` / `attributes.llm.output_messages` (output side). At span or trace granularity, `{conversation}` is treated as a regular template variable and resolved via column mappings like any other. ### Multi-evaluator tasks A task can contain evaluators at different granularities. At runtime the system uses the **highest** granularity (session > trace > span) for data fetching and automatically **splits into one child run per evaluator**. Per-evaluator `query_filter` in the task's evaluators JSON further narrows which spans are included (e.g., only tool-call spans within a session). --- ## Basic CRUD ### AI Integrations AI integrations store the LLM provider credentials the evaluator uses. For full CRUD — listing, creating for all providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Gemini, NVIDIA NIM, custom), updating, and deleting — use the **arize-ai-provider-integration** skill. Quick reference for the common case (OpenAI): ```bash # Check for an existing integration first ax ai-integrations list --space SPACE # Create if none exists ax ai-integrations create \ --name "My OpenAI Integration" \ --provider openAI \ --api-key $OPENAI_API_KEY ``` Copy the returned integration ID — it is required for `ax evaluators create --ai-integration-id`. ### Evaluators ```bash # List / Get ax evaluators list --space SPACE ax evaluators get ID # accepts name or ID ax evaluators get NAME --space SPACE # required when using name instead of ID ax evaluators list-versions NAME_OR_ID ax evaluators get-version VERSION_ID # Create (creates the evaluator and its first version) ax evaluators create \ --name "Answer Correctness" \ --space SPACE \ --description "Judges if the model answer is correct" \ --template-name "correctness" \ --commit-message "Initial version" \ --ai-integration-id INT_ID \ --model-name "gpt-4o" \ --include-explanations \ --use-function-calling \ --classification-choices '{"correct": 1, "incorrect": 0}' \ --template 'You are an evaluator. Given the user question and the model response, decide if the response correctly answers the question. User question: {input} Model response: {output} Respond with exactly one of these labels: correct, incorrect' # Create a new version (for prompt or model changes — versions are immutable) ax evaluators create-version NAME_OR_ID \ --commit-message "Added context grounding" \ --template-name "correctness" \ --ai-integration-id INT_ID \ --model-name "gpt-4o" \ --include-explanations \ --classification-choices '{"correct": 1, "incorrect": 0}' \ --template 'Updated prompt... {input} / {output} / {context}' # Update metadata only (name, description — not prompt) ax evaluators update NAME_OR_ID \ --name "New Name" \ --description "Updated description" # Delete (permanent — removes all versions) ax evaluators delete NAME_OR_ID ``` **Key flags for `create`:** | Flag | Required | Description | |------|----------|-------------| | `--name` | yes | Evaluator name (unique within space) | | `--space` | yes | Space name or ID to create in | | `--template-name` | yes | Eval column name — alphanumeric, spaces, hyphens, underscores | | `--commit-message` | yes | Description of this version | | `--ai-integration-id` | yes | AI integration ID (from above) | | `--model-name` | yes | Judge model (e.g. `gpt-4o`) | | `--template` | yes | Prompt with `{variable}` placeholders (single-quoted in bash) | | `--classification-choices` | yes | JSON object mapping choice labels to numeric scores e.g. `'{"correct": 1, "incorrect": 0}'` | | `--description` | no | Human-readable description | | `--include-explanations` | no | Include reasoning alongside the label | | `--use-function-calling` | no | Prefer structured function-call output | | `--invocation-params` | no | JSON of model params e.g. `'{"temperature": 0}'` | | `--data-granularity` | no | `span` (default), `trace`, or `session`. Only relevant for project tasks, not dataset/experiment tasks. See Data Granularity section. | | `--direction` | no | Optimization direction: `maximize` or `minimize`. Sets how the UI renders trends. | | `--provider-params` | no | JSON object of provider-specific parameters | ### Tasks > `PROJECT_NAME`, `DATASET_NAME`, and `evaluator_id` all accept a name or base64 ID. ```bash # List / Get ax tasks list --space SPACE ax tasks list --project PROJECT_NAME ax tasks list --dataset DATASET_NAME --space SPACE ax tasks get TASK_ID # Create (project — continuous) ax tasks create \ --name "Correctness Monitor" \ --task-type template_evaluation \ --project PROJECT_NAME \ --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \ --is-continuous \ --sampling-rate 0.1 # Create (project — one-time / backfill) ax tasks create \ --name "Correctness Backfill" \ --task-type template_evaluation \ --project PROJECT_NAME \ --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \ --no-continuous # Create (experiment / dataset) ax tasks create \ --name "Experiment Scoring" \ --task-type template_evaluation \ --dataset DATASET_NAME --space SPACE \ --experiment-ids "EXP_ID_1,EXP_ID_2" \ # base64 IDs from `ax experiments list --space SPACE -o json` --evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"output": "output"}}]' \ --no-continuous # Trigger a run (project task — use data window) ax tasks trigger-run TASK_ID \ --data-start-time "2026-03-20T00:00:00" \ --data-end-time "2026-03-21T23:59:59" \ --wait # Trigger a run (experiment task — use experiment IDs) ax tasks trigger-run TASK_ID \ --experiment-ids "EXP_ID_1" \ # base64 ID from `ax experiments list --space SPACE -o json` --wait # Monitor ax tasks list-runs TASK_ID ax tasks get-run RUN_ID ax tasks wait-for-run RUN_ID --timeout 300 ax tasks cancel-run RUN_ID --force ``` **Time format for trigger-run:** `2026-03-21T09:00:00` — no trailing `Z`. **Additional trigger-run flags:** | Flag | Description | |------|-------------| | `--max-spans` | Cap processed spans (default 10,000) | | `--override-evaluations` | Re-score spans that already have labels | | `--wait` / `-w` | Block until the run finishes | | `--timeout` | Seconds to wait with `--wait` (default 600) | | `--poll-interval` | Poll interval in seconds when waiting (default 5) | **Run status guide:** | Status | Meaning | |--------|---------| | `completed`, 0 spans | The eval index lags 1–2 hours — spans ingested recently may not be indexed yet. Shift the window to data at least 2 hours old, or widen the time range to cover more historical data. | | `cancelled` ~1s | Integration credentials invalid | | `cancelled` ~3min | Found spans but LLM call failed — check model name or key | | `completed`, N > 0 | Success — check scores in UI | --- ## Workflow A: Create an evaluator for a project