-
Treat default table columns as the minimum Phoenix integration, not the target experience.
-
Design telemetry specifically to activate Phoenix’s strongest visual features: embedding projections, clustering, trace trees, session views, annotations, comparisons, metrics, and cost analysis.
-
Log vector embeddings alongside the associated text inputs and outputs so Phoenix can generate UMAP point-cloud visualizations.
-
Attach meaningful dimensions to embedding records so point clouds can be colored and segmented by tags, correctness, prediction outcome, experiment group, or other diagnostic attributes.
-
Provide both primary inference data and reference baseline data when semantic drift analysis is required.
-
Structure reference and primary datasets consistently so Phoenix can identify, cluster, and rank regions of semantic drift.
-
Assign a stable session ID to every related group of traces.
-
Use session IDs to group multi-turn conversations, asynchronous workflows, and related operations into a unified Phoenix session view.
-
Preserve the real execution hierarchy through correct parent-child span relationships.
-
Represent agentic execution using meaningful span kinds and nesting, such as:
Agent -> Chain -> Tool -> LLM
-
Never emit related spans as an unrelated flat list when their execution hierarchy is known.
-
Follow the OpenInference semantic conventions for span kinds, attributes, inputs, outputs, model data, and relationships.
-
Log the prompt template separately from the rendered prompt sent to the model.
-
Log prompt variables as structured attributes rather than embedding them only inside the final prompt text.
-
Preserve enough prompt information for Phoenix to reconstruct how each variable produced the rendered prompt.
-
Attach structured metadata and tags to spans for every dimension likely to support filtering, comparison, diagnosis, or experimentation.
-
Prefer stable, consistently named metadata fields such as:
experiment_groupmodeltemperatureenvironmenttenantworkflowfeaturereleasedatasetuser_feedback
-
Do not encode important diagnostic dimensions only inside unstructured span text.
-
Record evaluator results using Phoenix’s native annotation schema rather than representing evaluators only as ordinary spans.
-
Attach evaluation annotations directly to the trace or span being evaluated.
-
Use clear annotation names and typed values, such as:
hallucination = falserelevance = 0.95correctness = 1quality = "pass"
-
Distinguish automated evaluator annotations from human feedback.
-
Preserve human-in-the-loop labels such as approval, rejection, thumbs-up, thumbs-down, correction, and reviewer notes.
-
Treat human feedback as ground-truth data that must remain queryable, filterable, and visually distinguishable.
-
Log prompt-token and completion-token counts using Phoenix-compatible OpenInference attributes.
-
Record the model identity accurately so Phoenix can associate usage with the correct pricing and performance characteristics.
-
Ensure spans have accurate start times, end times, and completion states.
-
Mark failures using proper error status and structured error attributes rather than placing failure information only in logs or output text.
-
Preserve latency at every meaningful level of execution, including agent, chain, retrieval, tool, and LLM spans.
-
Make retrieval operations independently observable when retrieval quality matters.
-
Log retrieved documents, relevance scores, ranks, identifiers, and source metadata in structured form.
-
Link evaluations, annotations, embeddings, retrieval records, prompts, and model calls back to the exact trace and span that produced them.
-
Use consistent schemas across environments and releases so Phoenix comparisons remain valid over time.
-
Optimize instrumentation for exploration, not merely ingestion: every important operational or quality question should correspond to a filterable field, visible annotation, trace relationship, embedding dimension, or metric.
-
Consider the Phoenix UI incomplete until an operator can:
- Follow a request through the complete trace tree.
- Group related traces into a session.
- Inspect prompt templates and substitutions.
- Compare successful and failed executions.
- Filter by operational and experimental metadata.
- See evaluator and human annotations directly on spans.
- Explore semantic clusters and drift.
- Analyze latency, errors, token usage, and cost.
- Link every diagnostic signal back to the exact execution that produced it.
