Menu
Mandatory requirements for developing evals in Phoenix Arize. The primary purpose of Phoenix is VISUAL consumption of information by people.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
The review lens for judging any code, seal, or story against the storage architecture's caliber — not "did it pass the meter" but "is this the best it can be." Use when QA'ing a sealed task, assessing a story, or judging whether built code rises to decisions.md and the storage architecture. Not a checklist pass; a wisdom pass.
Ideal target-state file map for KyzoDB — where each seat lives and what it owns. Use before placing a construct, naming a file, choosing a zone, or judging whether code sits in the wrong place. Not current-tree inventory (read the tree) and not per-construct coding law (rust-* skills).
Create precise, high-context direction for LLMs as Q/A/E triplets — the hardest question, the ruled answer, the enforcement that makes it stick. Use when writing specs, rulings, migration destinies, agent priming, review standards, or any context where an LLM must make a choice exactly the way you would.
Gate whether a working set of atoms/clusters is closed under dependency edges (OPEN lists crossings). Use after organize-work when the border must be a real min-cut. Not organize-work. Not architecture-design.
Hoist all reads to session entry and sink all writes to session exit around a pure Decision middle. Use when a host/session path mixes IO with meaning. Not name-and-seat-construct. Not organize-work.
Name a construct's kind of truth and seat it in exactly one zone (KEEP or SPLIT). Use when placing or splitting a type/module/bag under zone law and future pressure. Not io-inversion. Not architecture-design. Not organize-work.
| name | phoenix-arize |
| description | Mandatory requirements for developing evals in Phoenix Arize. The primary purpose of Phoenix is VISUAL consumption of information by people. |
Treat default table columns as the minimum Phoenix integration, not the target experience.
Design telemetry specifically to activate Phoenix’s strongest visual features: embedding projections, clustering, trace trees, session views, annotations, comparisons, metrics, and cost analysis.
Log vector embeddings alongside the associated text inputs and outputs so Phoenix can generate UMAP point-cloud visualizations.
Attach meaningful dimensions to embedding records so point clouds can be colored and segmented by tags, correctness, prediction outcome, experiment group, or other diagnostic attributes.
Provide both primary inference data and reference baseline data when semantic drift analysis is required.
Structure reference and primary datasets consistently so Phoenix can identify, cluster, and rank regions of semantic drift.
Assign a stable session ID to every related group of traces.
Use session IDs to group multi-turn conversations, asynchronous workflows, and related operations into a unified Phoenix session view.
Preserve the real execution hierarchy through correct parent-child span relationships.
Represent agentic execution using meaningful span kinds and nesting, such as:
Agent -> Chain -> Tool -> LLM
Never emit related spans as an unrelated flat list when their execution hierarchy is known.
Follow the OpenInference semantic conventions for span kinds, attributes, inputs, outputs, model data, and relationships.
Log the prompt template separately from the rendered prompt sent to the model.
Log prompt variables as structured attributes rather than embedding them only inside the final prompt text.
Preserve enough prompt information for Phoenix to reconstruct how each variable produced the rendered prompt.
Attach structured metadata and tags to spans for every dimension likely to support filtering, comparison, diagnosis, or experimentation.
Prefer stable, consistently named metadata fields such as:
experiment_groupmodeltemperatureenvironmenttenantworkflowfeaturereleasedatasetuser_feedbackDo not encode important diagnostic dimensions only inside unstructured span text.
Record evaluator results using Phoenix’s native annotation schema rather than representing evaluators only as ordinary spans.
Attach evaluation annotations directly to the trace or span being evaluated.
Use clear annotation names and typed values, such as:
hallucination = falserelevance = 0.95correctness = 1quality = "pass"Distinguish automated evaluator annotations from human feedback.
Preserve human-in-the-loop labels such as approval, rejection, thumbs-up, thumbs-down, correction, and reviewer notes.
Treat human feedback as ground-truth data that must remain queryable, filterable, and visually distinguishable.
Log prompt-token and completion-token counts using Phoenix-compatible OpenInference attributes.
Record the model identity accurately so Phoenix can associate usage with the correct pricing and performance characteristics.
Ensure spans have accurate start times, end times, and completion states.
Mark failures using proper error status and structured error attributes rather than placing failure information only in logs or output text.
Preserve latency at every meaningful level of execution, including agent, chain, retrieval, tool, and LLM spans.
Make retrieval operations independently observable when retrieval quality matters.
Log retrieved documents, relevance scores, ranks, identifiers, and source metadata in structured form.
Link evaluations, annotations, embeddings, retrieval records, prompts, and model calls back to the exact trace and span that produced them.
Use consistent schemas across environments and releases so Phoenix comparisons remain valid over time.
Optimize instrumentation for exploration, not merely ingestion: every important operational or quality question should correspond to a filterable field, visible annotation, trace relationship, embedding dimension, or metric.
Consider the Phoenix UI incomplete until an operator can: