optimize-trace-observability

name	optimize-trace-observability
description	Improve Coval trace quality after basic ingestion works. Use when traces are sparse, missing useful STT/LLM/TTS/tool spans, missing attributes needed for Coval built-in metrics, or when a customer wants maximum debugging and observability value from agent traces.

Optimize Coval Trace Observability

Turn a working but thin Coval trace into a useful debugging artifact. Prefer to inspect a proven Coval trace first. If setup-tracing has launched an initial asynchronous validation run that is still pending, use the waiting time to make safe code-visible enrichment, then re-check the finished run before declaring the optimization complete.

Read First

Load these references as needed:

• ../references/span-schema.md for canonical spans, attributes, aliases, and guardrails
• ../references/agent-type-routing.md for framework-specific trace boundaries
• ../references/coval-tracing-reference.md for viewer/search behavior and ingestion limits
• ../references/vapi-artifact-tracing.md when the agent uses Vapi webhooks

Phase 1: Inspect Current Trace Quality

Start from evidence, not assumptions.

1. Find one recent traced simulation or conversation in Coval. Use the Coval CLI/API or Trace Search instead of asking the user for a screenshot when credentials are available.
2. If the only candidate is an in-flight validation run from setup-tracing, start or continue a bounded CLI/API poll loop and do not block idly. While waiting, inspect the code path and add only enrichment that is clearly safe from the implementation.
3. Inspect the trace viewer or exported trace dump once trace data exists.
4. Classify the current trace:
   • no trace
   • trace exists but only root/provider spans
   • one span type dominates (for example, nearly all rows are conversation), which usually means child spans were not emitted or were named too broadly
   • STT/LLM/TTS spans exist but lack attributes
   • tool spans missing
   • parent/child structure is flat or misleading
   • attributes are unsafe, oversized, or high-cardinality
   • turn spans exist but are undifferentiated (all named turn with no role distinction) — this makes the trace viewer a wall of identical rows and hides the conversation structure; see Phase 2 for the fix
5. Check whether existing framework instrumentation should be enriched instead of duplicated.

Do not add manual duplicate spans for operations already emitted by Pipecat, LiveKit, Vapi, or an existing OTel integration unless the existing span cannot be enriched. For Vapi-hosted PSTN agents, be precise about what the webhook can prove: tool-calls webhooks are real tool execution; end-of-call-report.artifact messages are real transcript/turn source data; provider-internal STT/LLM/TTS timing is not available unless Vapi explicitly provides it.

Phase 1b: Discover Business Events Already In The Code

Before deciding what to add, grep the agent code for business-event surface area the customer already implements but isn't tracing. Search for terms like cart, order, intent, tool_call, function_call, handoff, escalat, payment, checkout, confirm, cancel, transfer, submit, dispatch, end_of_call, conversation_end, system_notify, webhook, event_name. Each match is a candidate for a business.<event> span or llm_tool_call span with a numeric attribute that can later become a customer-signal metric. This is the most common missed coverage layer: the customer's protocol already defines meaningful business events that never make it into traces because the first pass only instrumented the audio/LLM pipeline.

Propose one span per distinct business event with at least one numeric attribute (cart total, item count, payment amount, escalation level, etc.). Also identify the single most important vertical-specific failure mode and add a metric-ready root attribute for it, for example roadside.dispatch.latency_ms, reservation.date.changed, identity.verification.completed, payment_plan.blocked, or handoff.required. Do not wait for the customer to ask "is that all?" before adding these.

Phase 2: Add Coval-Native Span Coverage

Prioritize spans that make Coval trace UI, built-in trace metrics, and custom trace metrics more useful:

• conversation: full call/session root
• turn: one user/assistant exchange when the framework exposes a turn boundary
• stt: final speech recognition result
• stt.provider.<name>: each STT provider attempt or fallback
• llm: model call
• tts: synthesis call
• llm_tool_call: tool/function execution
• vad: speech activity decisions when relevant
• pipeline or transport: only when they help diagnose routing/audio issues

Keep span names stable and low-cardinality. Put IDs, provider names, endpoint names, and dynamic details in attributes. Match the public span naming convention before adding custom business spans: canonical names get semantic colors, labels, and built-in trace metric support.

For Vapi artifact-based enrichment:

• create turn.assistant and turn.user spans (not a flat turn) from artifact.messages timing windows — role-differentiated names produce visually distinct rows in the trace viewer; add turn.text (first ~200 chars of the message) so content is readable without opening the detail panel
• create llm_tool_call spans from live tool-call webhooks and handler results
• use provider marker spans only when clearly labeled with trace.timing=metadata_marker, or use stt_marker/llm_marker/tts_marker
• never give marker spans the same full duration as the enclosing turn
• never add fake confidence, fake TTFB, or fake provider latency values
• when reconstructing turn spans from Vapi artifact.messages, apply the timestamp guardrails in ../references/vapi-artifact-tracing.md to avoid OTel int64 overflow from the time field and misrouted secondsFromStart values

Prefer OTel span events over new spans for moment-in-time annotations. A span event (span.add_event("simulation_id_received", {...})) is a cheap timestamped marker on an existing span — it does not bloat the span count, does not affect trace metrics, and gives the trace viewer a visible flag on the parent timeline. Use events for milestones like simulation_id_received, first_inbound_audio, first_speech_detected, cart_sent, tool_dispatched, websocket_disconnect, and conversation_end on the conversation root or the relevant turn. Reserve full spans for things with non-trivial duration or parent/child structure.

Phase 3: Add High-Value Attributes

Use the canonical attributes from ../references/span-schema.md.

Minimum valuable set:

• stt: transcript, metrics.ttfb, stt.confidence when available
• stt.provider.<name>: stt.providerName, metrics.ttfb, stt.confidence, error status when a provider fails
• llm: metrics.ttfb, llm.finish_reason, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, model/provider metadata when available
• tts: metrics.ttfb, provider/voice metadata when safe
• llm_tool_call: function.name, tool_call_id, function.arguments if safe and bounded, tool.latency_ms, numeric tool.error, numeric tool.dependency_unavailable, tool.result.count when applicable
• conversation: tool.call.count, tool.failure.count, numeric workflow.completed, numeric workflow.dependency_blocked, numeric workflow.fallback_used when the session boundary is available
• vertical-specific root attributes: one or two numeric signals that describe the customer's main workflow risk, such as roadside.dispatch.latency_ms, roadside.dispatch.slow, fnol.fields_captured.count, fraud.pattern_detected, reservation.date.changed, identity.verification.completed, or payment_plan.blocked
• conversation aggregates when safely available: call.cost_usd, conversation.summary.length_chars, transcript.length_chars, transcript.turn.count, and per-role turn counts
• custom spans: one numerical attribute that can become a custom trace metric, such as duration_ms, retry_count, confidence_score, queue_wait_ms, or external_api_latency_ms

Set OTel status to ERROR on failing provider/tool/API spans. Coval custom trace metrics can calculate error_rate and success_rate from span status.

Also emit numeric 0/1 flags for important rates. Some public metric creation APIs require a numeric metric_attribute; average over these flags preserves the rate while still working in those environments. For Vapi-hosted agents, prefer tool/workflow attributes such as tool.error, tool.latency_ms, tool.call.count, workflow.completed, workflow.dependency_blocked, and workflow.fallback_used over provider latency attributes unless the provider timing is explicitly measured.

Phase 4: Protect Customers

Before committing enrichment, remove or bound:

• API keys, tokens, passwords, session cookies, account secrets, and credentials
• raw audio blobs or base64 audio
• full prompts/responses when they may contain PII
• unbounded transcripts in custom attributes unrelated to stt.transcript
• high-cardinality span names
• duplicated successful export retries

Prefer summaries and counts:

• tool.result.count
• tool.result.success
• tool.error
• tool.call.count
• tool.failure.count
• workflow.completed
• workflow.dependency_blocked
• workflow.fallback_used
• prompt.message_count
• response.length
• http.status_code
• retry_count

Phase 5: Verify Improved Value

After changes:

1. Run one representative simulation or conversation through the Coval CLI/API, or reuse the in-flight validation run if it exercises the changed code.
2. Poll through the CLI/API until the run finishes and trace data appears. While it is pending, prepare candidate metrics or documentation notes instead of waiting idle.
3. Open the Coval trace viewer.
4. Confirm the trace has meaningful hierarchy and expected span colors.
5. Check Trace Search can filter by span name, status, duration, provider, or attributes.
6. If new numerical attributes were added, run configure-trace-metrics to create at least one metric against them after the span/attribute is visible in real trace data.

Report before/after differences in concrete terms, such as span count, new span names, new attributes, and the specific debugging question the trace can now answer.