ops-eng-o2r-phoenix

name

ops-eng-o2r-phoenix

description

Use when working with the Arize Phoenix backend for otel-a2a-relay (o2r). Names the specific Phoenix UI surface for each question (waterfall vs JSON tree vs session list vs agent topology), with direct URLs and click paths. Triggers - phoenix, arize phoenix, o2r phoenix, phoenix sessions, phoenix trace, where is the waterfall, agent graph, openinference, o2r-harness, o2r-view, o2r-phoenix-bootstrap.

Phoenix backend for o2r

Local Phoenix runs at http://localhost:6006. The relay points OTEL_EXPORTER_OTLP_ENDPOINT at the same host (Phoenix accepts OTLP/HTTP on 6006).

Repo: otel-a2a-relay/arize_phoenix/. Upstream: Arize Phoenix.

Which UI surface answers which question

Default to visual views, not the JSON tree. The JSON tree on the trace detail page is a fallback - it is not the waterfall.

• "Show me one session, all hops" - Sessions tab: http://localhost:6006/projects/default/sessions. One row per session.id; the entry point for any o2r demo run.
• "Show me the visual waterfall for one trace" - click a trace from the Sessions row; the Trace detail page renders the timeline by default. JSON means you clicked the wrong tab.
• "Show me the agent topology" - Agent Graph tab. Nodes are agents (agent.role + agent.specialization); edges from graph.node.parent_id (A -> B outbound, B -> A return).
• "Show me LLM calls inside a hop" - LLM-kind spans nest under AGENT-kind roots in the Trace waterfall. OpenInference attributes (openinference.span.kind = LLM) drive the rendering.
• "Show me datasets / annotations" - Datasets and Annotations sections in the left nav. o2r-phoenix-bootstrap provisions relay-decisions-golden, relay-failures-regression, relay_failure_class, task_outcome_correct.

Operate

From otel-a2a-relay/ workspace root:

uv sync --all-packages
make phoenix-fg              # Phoenix in foreground (operator-owned, separate terminal)
make phoenix-bootstrap       # idempotent: annotation configs + datasets
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:6006 make luca-demo
open http://localhost:6006   # lands on home, click Sessions

Phoenix elsewhere (remote host): OTEL_EXPORTER_OTLP_ENDPOINT=http://phoenix.local:6006 uv run o2r-harness.

CLIs the relay ships

• o2r-harness - posts the worked-example trace, prints session.id and the validation checklist.
• o2r-view - reduces a session's spans to a readable per-hop log (waterfall in text form for mobile / dictation). Also drives make gif CTX=<context>.
• o2r-phoenix-bootstrap - idempotently provisions annotation configs + datasets via Phoenix REST.

Harness validation checklist

After o2r-harness, walk the Sessions / Agent Graph / Trace-tree checklist in docs/harness.md. If any view fails to render, don't start relay implementation - open an issue with screenshots.

If any of those fails to render, do not start relay implementation. Open an issue with screenshots.

Common traps

• JSON tree is not the waterfall. The trace detail page has a JSON view and a timeline view. The timeline is the waterfall. If Kai asks "where is the visual view," answer Trace detail -> timeline tab, not the default JSON dump.
• DB stuck on alembic error. ~/.phoenix/phoenix.db migration fails across version bumps with ResolutionError. Fix: rm ~/.phoenix/phoenix.db and restart. The harness reseeds in seconds.
• No spans showing. Check OTEL_EXPORTER_OTLP_ENDPOINT actually points at :6006 (Phoenix), not :4318 (Tempo's port). Same protocol, different backend.
• OpenInference attributes missing. Agent Graph stays empty without openinference.span.kind and graph.node.parent_id. The relay sets these; bespoke instrumentation must too.