telemetry-analyzer

name	telemetry-analyzer
description	Use when the user wants to analyze agent telemetry traces to find bugs and get fix recommendations — walks through exporting traces from a local or remote watsonx Orchestrate server, parsing raw OTel JSON or Langfuse-format trace JSON directly, and reasoning over them to identify failures and suggest fixes.

Agent Telemetry Analyzer

This skill exports agent telemetry traces from watsonx Orchestrate and analyzes them directly from raw OTel JSON or Langfuse-format trace JSON to produce structured bug reports with root-cause analysis and fix recommendations.

Two trace formats are supported:

• OTel JSON — produced by the classic TracesController.export_trace_to_json() ADK path and local IBM telemetry servers. Top-level key: traceData.resourceSpans.
• Langfuse JSON — produced by the agentops-v3 REST API (GET /v1/agentops-v3/traces/<id>) and the export_traces_agentops_v3.py script. Top-level key: observations (array of GENERATION / CHAIN / SPAN objects).

Always detect the format first before reading any span data. See the detect_format() helper in Step 3C and the normalization table in Step 4.

Setup: See README.md for prerequisites (ADK installation, .env configuration, and environment activation) before running any scripts.

What this skill does

1. Identifies the trace source — remote hosted env (wxO Traces API or agentops-v3 REST API), local env with IBM telemetry, or JSON files already on disk (OTel or Langfuse format)
2. Exports trace files — discovers traces by agent name or ID, paginates through results, and downloads them as OTel JSON (via ADK TracesController) or Langfuse JSON (via export_traces_agentops_v3.py / search_traces_agentops_v3.py)
3. Analyzes for bugs — detects the JSON format, normalizes spans to a common structure, and scans for hard errors, LLM failures, tool call failures, agent logic bugs, flow issues, token anomalies, and cache efficiency
4. Produces a bug report — structured HTML report with an executive summary, trace summary table, critical issues, warnings, observations, and fix recommendations (multi-trace analysis only — a single-trace request goes straight to the conversational flow report)
5. Produces conversational flow reports — thread-scoped reports that reconstruct multi-turn conversations and surface per-turn failures

When to use this skill

Use this skill when you need to:

• Debug a failing or misbehaving watsonx Orchestrate agent
• Understand why an agent returned an empty or incorrect response
• Trace LLM token usage and identify prompt bloat or context overflow
• Identify tool call failures, retry loops, or hung async flows
• Analyze a batch of traces and surface the most impactful issues
• Reconstruct a full multi-turn conversation from thread-scoped traces

How to invoke

Ask questions like:

• "Analyze the telemetry for the My Agent traces from the last 20 minutes"
• "What went wrong in trace f492e71b957ccec2d07096ae99395d19?"
• "Export and inspect all traces for the Supervisor agent in the last 2 hours"
• "Why did this agent return an empty response?"
• "Show me the conversational flow for this trace"

Command-line usage

Standard scripts (_adk.py) — wxO Traces API / local IBM telemetry

# Search traces and save IDs for bulk export
python scripts/search_traces_adk.py --agent-name "My Agent" --last 1h --all \
    --save-ids data/trace_ids.txt

# Export all discovered traces
python scripts/export_traces_adk.py --ids-file data/trace_ids.txt

# Export a single trace by ID
python scripts/export_traces_adk.py --trace-id <32-char-hex-id>

agentops-v3 scripts (_agentops_v3.py) — agentops-v3 REST API

# Search traces for an agent (last 20 minutes by default)
python scripts/search_traces_agentops_v3.py --last 20m

# Narrow or widen the time window
python scripts/search_traces_agentops_v3.py --last 2h
python scripts/search_traces_agentops_v3.py --start 2025-01-01T09:00:00Z --end 2025-01-01T21:00:00Z

# Collect every trace in the window and save IDs for bulk export
python scripts/search_traces_agentops_v3.py --last 1h --all \
    --save-ids data/trace_ids.txt

# Quick count before committing to a full export
python scripts/search_traces_agentops_v3.py --last 2h --count

# Export a single trace by ID
python scripts/export_traces_agentops_v3.py --trace-id <32-char-hex-id>

# Bulk-export from a saved ID list
python scripts/export_traces_agentops_v3.py --ids-file data/trace_ids.txt

Analysis capabilities

Trace export and discovery

• Cursor-based pagination through the wxO Traces API with automatic 429 retry
• Supports --limit N, --all, and --count modes for controlling result volume
• Saves trace files as data/<trace_id>.json — OTel format via export_traces_adk.py, Langfuse format via export_traces_agentops_v3.py
• Works with remote hosted environments and local servers started with --with-ibm-telemetry
• search_traces_agentops_v3.py / export_traces_agentops_v3.py use the raw agentops-v3 REST API (bearer token from active env, no ADK SDK calls)

Bug detection

Detected from normalized spans, supporting both OTel (traceData.resourceSpans[0].scopeSpans[0].spans) and Langfuse (observations[]) formats:

• Hard errors — any span with status.code == STATUS_CODE_ERROR
• LLM failures — finish_reason == "length", empty answer.task output, zero output tokens (via llm.token_count.completion / gen_ai.usage.output_tokens attributes)
• Tool call failures — empty ToolMessage content in *.tool spans, retry loops, duration_ms > 30 000 ms
• Agent logic bugs — missing answer.task span, empty final answer, repeated agent.task cycles (multi-step loop), empty collaborator output
• Flow issues — has_error on any span, widget_result.task or collaborator.task with empty output
• Token anomalies — cumulative token totals exceeding 50 000, high input:output token ratio
• Cache efficiency — total cache-read tokens, cache hit rate below 20 % on multi-turn threads, zero cache reads across ≥ 3-turn threads, cache-write/read imbalance

Conversational flow analysis

• Detects trace format and extracts thread_id from the correct location:
  • OTel: traceloop.association.properties.thread_id or thread.id span attribute
  • Langfuse: metadata.attributes.thread_id or sessionId top-level field
• Searches the local trace cache and the API for all traces belonging to a thread
• Reconstructs user message, routing decision, collaborator steps, and final response per turn
• Flags turns where the agent returned no response or encountered an error

Output format

The analyzer produces:

1. Bug Report (HTML artifact for 3+ issues, inline for fewer)

   • Report header with agent ID, name, environment, date, and UTC time window
   • Executive Summary with KPI stat cards (for 10+ traces)
   • Categorical trace summary table grouped by routing pattern or error type
   • Per-trace detail table with start time, span count, token counts, and status
   • 🔴 Critical Issues, 🟠 Warnings, and 🟡 Observations tables with affected trace IDs
   • 🛠️ Fix Recommendations with root cause and actionable steps per issue

2. Conversational Flow Report (HTML artifact, thread-scoped)

   • Thread metadata: thread ID, turn count, agent, environment, time window, overall outcome
   • Per-turn timeline: user message, routing decision, collaborator steps, response delivered, token counts, error flags
   • Full span timing breakdown across all turns in the thread
   • Root cause summary comparing what worked vs. what failed

Supporting files

Scripts in scripts/ (created on first use by Step 1 if not already present):

Script	Output format	Auth mechanism	Notes
search_traces_adk.py	OTel	ADK TracesController	Cursor-based pagination, agent-name filter, rate-limit retry
export_traces_adk.py	OTel	ADK TracesController	Single or bulk export by trace ID
search_traces_agentops_v3.py	Langfuse	Bearer token from active env	Page-based pagination, returns all traces in time window
export_traces_agentops_v3.py	Langfuse	Bearer token from active env	GET /v1/agentops-v3/traces/<id>, no ADK SDK trace call

Use the *_agentops_v3.py scripts when the environment uses the agentops-v3 REST API. Use the standard scripts for environments backed by the classic wxO Traces API.

---

Step-by-step workflow

Step 1: Verify helper scripts are present

File placement rules — apply throughout all steps:

• All Python scripts written to disk must be saved under scripts/ (e.g. scripts/my_helper.py).
• All output files (.json trace dumps, .txt ID lists, etc.) must be saved under data/ (e.g. data/trace_ids.txt, data/<trace_id>.json).
• Inline code snippets shown in this document are for reading and reasoning only — they are not written to disk unless explicitly instructed.

Before doing anything else, confirm the required scripts exist on disk:

scripts/search_traces_adk.py
scripts/export_traces_adk.py
scripts/search_traces_agentops_v3.py
scripts/export_traces_agentops_v3.py

If any are missing, use write_file to create them — the canonical source for each script is in the scripts/ directory of this skill. Write each missing file to its correct path under scripts/ before continuing to Step 2.

---

Step 2: Determine the trace source

If the user's request does not clearly state where the traces or agent reside, always use ask_followup_question to clarify before doing anything else. Do not assume a source or proceed to Step 3 without a confirmed answer to all of the following that are not already clear from context:

• Is the environment remote (a hosted wxO instance) or local (a locally running server)?
• If remote — what is the environment name (e.g. the name used with orchestrate env activate)?
• If the user mentioned an agent — what is the agent name or ID?
• What time window should be searched (or should it default to the last 20 minutes)?

Once the source is confirmed, classify it as one of:

1. Remote env — a hosted watsonx Orchestrate environment activated with orchestrate env activate. Always try the _agentops_v3.py scripts first; fall back to the _adk.py scripts if the agentops-v3 search returns an error or zero traces. See Step 3A for the full fallback procedure.
2. Local env — IBM telemetry profile — a locally running orchestrate server started with --with-ibm-telemetry. The Traces API is available at http://localhost:4321. Use the _adk.py scripts directly — agentops-v3 is not available on local environments.
3. Trace files on disk — the user already has one or more {trace_id}.json files on disk ready to analyze. Confirm they are in the data/ directory; if not, ask the user for the path before proceeding.

---

Step 3: Export the trace files

Option A — Remote env (agentops-v3 first, ADK fallback)

For all remote environments, always try _agentops_v3.py first. Fall back to _adk.py only if the agentops-v3 search fails or returns zero traces.

Step A1 — Try agentops-v3:

# 1. Activate the target environment
orchestrate env activate <env-name>

# 2. Search traces via agentops-v3
python scripts/search_traces_agentops_v3.py \
    --start 2025-01-01T09:00:00Z --end 2025-01-01T21:00:00Z \
    --all --save-ids data/trace_ids.txt

If the command exits with a non-zero status or prints Found 0 trace(s), proceed to Step A2. Otherwise, export the discovered traces:

# 3. Export all discovered traces
python scripts/export_traces_agentops_v3.py --ids-file data/trace_ids.txt

Step A2 — Fall back to ADK (only if agentops-v3 failed or returned 0 traces):

# 2. Search traces via ADK — IDs checkpointed to disk after every page
python scripts/search_traces_adk.py --agent-name "My Agent" \
    --start 2025-01-01T09:00:00Z --end 2025-01-01T21:00:00Z \
    --all --save-ids data/trace_ids.txt

# 3. Export all discovered traces
python scripts/export_traces_adk.py --ids-file data/trace_ids.txt

search_traces_adk.py uses cursor-based pagination, retries automatically on 429 rate-limit responses, and handles mid-run 401 token expiry by refreshing the client. Always pass --save-ids so trace IDs are checkpointed to disk page-by-page and are not lost on timeout.

Use --limit N (default 50) for a capped sample, --all to exhaust the full window, or --count for a quick volume figure before committing to a full export.

Option B — Local env (IBM telemetry profile)

# 1. Start the local server with IBM telemetry enabled
orchestrate server start --with-ibm-telemetry --accept-terms-and-conditions

# 2. Activate the local environment
orchestrate env activate local

# 3. Search and export — identical to Option A
python scripts/search_traces_adk.py --agent-name "My Agent" --last 30m --all \
    --save-ids data/trace_ids.txt
python scripts/export_traces_adk.py --ids-file data/trace_ids.txt

The local Traces API endpoint is http://localhost:4321. The scripts detect is_local_dev() automatically and set service_names=["wxo-server"] in the filter, which is required when FORCE_SINGLE_TENANT=true.

Option C — Trace files already on disk

Confirm the directory or file path with the user. Always detect the format before reading spans. Use this helper:

import json

def detect_format(raw: dict) -> str:
    """Return 'otel' or 'langfuse'."""
    if isinstance(raw.get('traceData'), dict) and 'resourceSpans' in raw['traceData']:
        return 'otel'
    if 'resourceSpans' in raw:  # traceData sometimes omitted at root
        return 'otel'
    if 'observations' in raw:   # Langfuse / agentops-v3 REST response
        return 'langfuse'
    return 'langfuse'           # safe default for agentops-v3 output

def get_spans(raw: dict) -> list:
    """Return a flat list of span/observation dicts regardless of format."""
    fmt = detect_format(raw)
    if fmt == 'otel':
        td = raw.get('traceData') or raw
        return [
            s
            for rs in td.get('resourceSpans', [])
            for ss in rs.get('scopeSpans', [])
            for s in ss.get('spans', [])
        ]
    return raw.get('observations', [])

Load a file and get its spans:

with open("data/<trace_id>.json") as f:
    raw = json.load(f)
fmt   = detect_format(raw)
spans = get_spans(raw)
print(f"Format: {fmt}, span/observation count: {len(spans)}")

If the user provides a directory, glob for *.json files and skip any files not matching the 32-character hex trace ID pattern.

---

Step 4: Analyze raw spans for bugs

Single-trace shortcut: If the user has asked to analyze exactly one trace (by ID or as a single file on disk), skip Steps 4 and 5 entirely. Jump straight to Step 7 to produce the conversational flow report for the thread that trace belongs to. Only return to Step 5 (bug report) if the user explicitly asks for one after seeing the flow report.

Load each trace file using detect_format() + get_spans() from Step 3C, then normalize every span with the adapter below before running any bug-detection logic.

4.1 Format normalization

from datetime import datetime, timezone

def normalize_span(s: dict, fmt: str) -> dict:
    """
    Return a uniform dict with these keys regardless of source format:
      name, start_utc, duration_ms, status_error (bool),
      input, output, attrs (dict: key -> scalar value)
    """
    if fmt == 'otel':
        start_ns  = int(s.get('startTimeUnixNano', 0))
        end_ns    = int(s.get('endTimeUnixNano', 0))
        start_utc = datetime.fromtimestamp(start_ns / 1e9, tz=timezone.utc)
        duration_ms = (end_ns - start_ns) / 1e6
        status_error = s.get('status', {}).get('code') in ('STATUS_CODE_ERROR', 2)
        attrs = {}
        for a in s.get('attributes', []):
            v = a['value']
            attrs[a['key']] = (
                v.get('stringValue') or v.get('intValue') or
                v.get('doubleValue') or v.get('boolValue')
            )
        input_val  = attrs.get('traceloop.entity.input')
        output_val = attrs.get('traceloop.entity.output')
    else:  # langfuse
        start_utc = datetime.fromisoformat(
            s.get('startTime', '1970-01-01T00:00:00Z').replace('Z', '+00:00'))
        end_str   = s.get('endTime') or s.get('startTime', '1970-01-01T00:00:00Z')
        end_dt    = datetime.fromisoformat(end_str.replace('Z', '+00:00'))
        duration_ms = (end_dt - start_utc).total_seconds() * 1000
        status_error = (s.get('level') == 'ERROR' or bool(s.get('statusMessage')))
        meta  = s.get('metadata') or {}
        attrs = dict(meta.get('attributes', {}))
        usage = s.get('usage') or {}
        if usage.get('input'):
            attrs['llm.usage.prompt_tokens'] = usage['input']
        if usage.get('output'):
            attrs['llm.usage.completion_tokens'] = usage['output']
        input_val  = s.get('input')
        output_val = s.get('output')

    return dict(
        name=s.get('name', ''),
        start_utc=start_utc,
        duration_ms=duration_ms,
        status_error=status_error,
        input=input_val,
        output=output_val,
        attrs=attrs,
    )

def get_attr(attrs: dict, key: str):
    """Read a scalar from the normalized attrs dict (works for both formats)."""
    return attrs.get(key)

Build the normalized span list at the start of every analysis:

fmt    = detect_format(raw)
spans  = get_spans(raw)
nspans = [normalize_span(s, fmt) for s in spans]

llm_spans  = [n for n in nspans if n['name'] == 'WatsonxChatModel.chat']
tool_spans  = [n for n in nspans if 'collaborator' in n['name'].lower()
               or n['attrs'].get('openinference.span.kind') == 'TOOL']

Field mapping reference — use this table when reading span data:

Concept	OTel key (in attrs after normalization)	Langfuse source (in attrs after normalization)
LLM input tokens	llm.token_count.prompt or llm.usage.prompt_tokens	llm.usage.prompt_tokens (from usage.input)
LLM output tokens	llm.token_count.completion or llm.usage.completion_tokens	llm.usage.completion_tokens (from usage.output)
LLM model	llm.model_name or gen_ai.request.model	llm.model_name or model top-level field
Thread ID	traceloop.association.properties.thread_id or thread.id	thread_id or thread.id
Finish reason	llm.finish_reason or gen_ai.usage.finish_reasons	llm.finish_reason
Cache read tokens	gen_ai.usage.cache_read_input_tokens or llm.token_count.cache_read	gen_ai.usage.cache_read_input_tokens or usage.cacheReadInputTokens (check raw obs)
Cache creation tokens	gen_ai.usage.cache_creation_input_tokens or llm.token_count.cache_creation	gen_ai.usage.cache_creation_input_tokens or usage.cacheCreationInputTokens (check raw obs)
Span/obs input	input (normalized field)	input (normalized field)
Span/obs output	output (normalized field)	output (normalized field)
LLM reasoning	inside output JSON → additional_kwargs.reasoning	inside output dict →

For each trace collect: start_utc, span names, duration_ms, status_error, collaborator tool call output, final answer output, thread_id (from attrs), and cache token counts.

Look for the following bug categories. For each issue found, record the span name, start_utc, and relevant attrs values.

4.2 Hard errors

• Any normalized span where status_error == True.
  • OTel source: status.code == "STATUS_CODE_ERROR" or 2.
  • Langfuse source: level == "ERROR" or statusMessage is non-empty.
• Capture span name, duration_ms, and the output field for the error message.

4.3 LLM failures

• get_attr(n['attrs'], 'llm.finish_reason') == "length" — context length exceeded (works for both formats after normalization).
• WatsonxChatModel.chat span with duration_ms == 0 or absent — silent LLM failure.
• Token counts: use llm.token_count.prompt / llm.token_count.completion (OTel attrs) or llm.usage.prompt_tokens / llm.usage.completion_tokens (Langfuse, auto-added by normalize_span). Zero completion tokens with non-zero prompt tokens = failed generation.
• Unexpectedly high prompt token count — possible system prompt bloat.
• Cache token counts: see §4.8. After normalization, try get_attr(attrs, 'gen_ai.usage.cache_read_input_tokens') first; fall back to the raw Langfuse usage dict. If absent on all LLM spans, note as an observation (provider may not support caching).

4.4 Tool / collaborator call failures

• Tool span / SPAN observation whose output field contains an empty or null content (i.e. ToolMessage.content == "").
  • Langfuse note: an empty content on the immediate chat_with_collaborator_* SPAN is the normal async dispatch pattern — the real result arrives via the nested collaborator CHAIN observation. Only flag as a failure if the outer collaborator CHAIN output is also empty.
• Any tool span with duration_ms > 30 000 ms — timeout or hung call.
• Same collaborator tool called twice in a row — retry loop.

4.5 Agent logic bugs

• No span/observation named answer (or answer.task in OTel) — agent never produced a final answer.
• answer span output is empty or whitespace-only.
• agent (or agent.task) span count > 2 — multi-step LLM loop (agent re-planning instead of delegating immediately).
• collaborator (or collaborator.task) span present but answer output is empty — collaborator returned nothing and supervisor silently swallowed it.
• Internal reasoning text present in answer output or output.additional_kwargs.reasoning before the user-facing reply — check both OTel and Langfuse paths.

4.6 Flow / widget issues

• Span/observation named widget_result (Langfuse) or widget_result.task (OTel) with empty output — widget state machine stalled.
• LangGraph GENERATION (Langfuse) or LangGraph.workflow span (OTel) with duration_ms >> sum of child span durations — unexplained gap in the workflow.

4.7 Token / cost anomalies

• Cumulative prompt + completion tokens across all WatsonxChatModel.chat spans exceeds 50 000.
• Prompt tokens >> completion tokens (ratio > 200:1) — system prompt dominates context, little room for conversation history.

4.8 Cache efficiency

Langfuse format caveat: Cache token attributes (cacheReadInputTokens, cacheCreationInputTokens) are not reliably collected in Langfuse-format traces. When the source format is Langfuse (fmt == 'langfuse'), attempt both the metadata.attributes and top-level usage fallback paths below. If both are absent after exhausting all fallbacks, set cache_unsupported = True for the trace and skip all cache-related analysis, stat cards, table columns, and observations for that trace. Do not flag missing cache attributes as a bug or observation for Langfuse-format traces — they are expected to be absent.

Extract per-WatsonxChatModel.chat normalized span, with a Langfuse usage dict fallback for cache fields that may not be hoisted into metadata.attributes:

def get_cache_tokens(nspan: dict, raw_obs: dict = None) -> tuple:
    """
    nspan   — normalized span dict (attrs already flattened from either format).
    raw_obs — original Langfuse observation dict (optional), used as fallback
              for cache fields that are in usage but not in metadata.attributes.
    Returns (cache_read, cache_creation) as ints.
    """
    attrs = nspan['attrs']
    cache_read = (
        get_attr(attrs, 'gen_ai.usage.cache_read_input_tokens') or
        get_attr(attrs, 'llm.token_count.cache_read') or 0
    )
    cache_creation = (
        get_attr(attrs, 'gen_ai.usage.cache_creation_input_tokens') or
        get_attr(attrs, 'llm.token_count.cache_creation') or 0
    )
    # Langfuse-specific fallback: check top-level usage dict
    if raw_obs and not cache_read and not cache_creation:
        usage = raw_obs.get('usage') or {}
        cache_read     = usage.get('cacheReadInputTokens') or 0
        cache_creation = usage.get('cacheCreationInputTokens') or 0
    return int(cache_read), int(cache_creation)

Aggregate across all LLM normalized spans in the trace (pass both the normalized span and the original raw observation when the source is Langfuse):

total_cache_read     = sum(get_cache_tokens(n, raw_obs=spans[i] if fmt=='langfuse' else None)[0]
                           for i, n in enumerate(llm_spans))
total_cache_creation = sum(get_cache_tokens(n, raw_obs=spans[i] if fmt=='langfuse' else None)[1]
                           for i, n in enumerate(llm_spans))

# For Langfuse traces: if both totals are still 0 after all fallbacks, cache data
# is not collected by this environment — mark the trace and skip cache analysis.
cache_unsupported = (fmt == 'langfuse') and (total_cache_read == 0) and (total_cache_creation == 0)

# Effective billable input = input_tokens - total_cache_read
# Cache hit rate = total_cache_read / (input_tokens - total_cache_creation)
#                 only meaningful when input_tokens > total_cache_creation > 0
denom = input_tokens - total_cache_creation
cache_hit_rate = (total_cache_read / denom) if (not cache_unsupported and denom > 0) else None

Flag the following — only when cache_unsupported is False:

• Low cache hit rate — cache_hit_rate < 0.20 (< 20 %) on a trace that belongs to a thread with ≥ 3 turns. The system prompt is likely not structured for prefix caching or varies per turn.
• Zero cache reads on multi-turn thread — every cache_read_input_tokens == 0 across ≥ 3 turns in the same thread. The cache is never being warmed or the prefix changes each turn.
• Cache write/read imbalance — total_cache_creation > total_cache_read * 5 across the batch. The cache is being populated but rarely re-used (cache TTL mismatch or non-repeating prefix).
• No cache attributes present (OTel only) — if the format is OTel and neither key is present on any LLM span, note it as an observation: the provider may not support prompt caching, or instrumentation does not emit these fields. Do not emit this observation for Langfuse-format traces.

---

Step 5: Produce the bug report (multi-trace analysis only)

Skip this step when the analysis scope is a single trace. Proceed directly to Step 7 instead.

Report title

Telemetry Bug Report — <AgentName>

Report header

Field	Value
Agent ID	<uuid>
Agent Name	<name>
Environment	Instance URL or local
Report Date	YYYY-MM-DD (UTC)
Time Window (UTC)	YYYY-MM-DD HH:MM – YYYY-MM-DD HH:MM UTC

Executive Summary (for 10+ traces)

Two rows of 4 KPI stat cards each:

• Row 1 — Total traces (note if sampled), failed count (.stat-bad), successful count (.stat-ok), time window duration.
• Row 2 — Total cache-read tokens, cache hit rate as %, total LLM input tokens, avg/p95 duration. If all traces in the batch are Langfuse format and cache_unsupported is True for every trace, omit the cache-read and cache hit rate stat cards entirely and render only a single row of 4 cards (total traces, failed, successful, avg/p95 duration).

No prose paragraphs — stat cards only.

Required CSS:

.stat-grid { display: grid; grid-template-columns: repeat(4, 1fr); gap: 10px; margin-bottom: 20px; }
.stat-card { background: #f7f8fa; border: 1px solid #e5e7eb; border-radius: 6px; padding: 12px 14px; }
.stat-val  { font-size: 22px; font-weight: 700; color: #1f2328; }
.stat-lbl  { font-size: 12px; color: #57606a; margin-top: 2px; }
.stat-bad  .stat-val { color: #b91c1c; }
.stat-ok   .stat-val { color: #15803d; }
.stat-warn .stat-val { color: #b45309; }

Cache-specific stat card values to compute before rendering:

# Exclude traces where cache data is known to be uncollected (Langfuse with no cache attrs)
cache_rows = [r for r in rows if not r.get("cache_unsupported")]

if cache_rows:
    total_cache_read_all     = sum(r.get("cache_read_tokens", 0)     for r in cache_rows)
    total_cache_creation_all = sum(r.get("cache_creation_tokens", 0) for r in cache_rows)
    total_input_cache        = sum(r.get("input_tokens", 0)          for r in cache_rows)

    denom_all = total_input_cache - total_cache_creation_all
    batch_cache_hit_rate = (total_cache_read_all / denom_all * 100) if denom_all > 0 else None

    cache_attrs_present = total_cache_read_all > 0 or total_cache_creation_all > 0
    hit_rate_display = f"{batch_cache_hit_rate:.1f}%" if cache_attrs_present else "N/A"
    hit_rate_class   = "stat-bad" if (cache_attrs_present and batch_cache_hit_rate is not None and batch_cache_hit_rate < 20) else "stat-ok"
    show_cache_cards = True
else:
    # All traces are Langfuse with no cache data — omit both cache stat cards
    show_cache_cards = False

Trace Summary section

Part 1 — Categorical summary table (group by routing pattern, error type, etc.):

Category	Traces	LLM Calls (avg)	Tool Calls (avg)	Success Rate	Status

Part 2 — Per-trace detail table (ordered by start time ascending):

If show_cache_cards is True (cache data available for at least some traces), use the full 7-column layout:

Trace ID	Start Time (UTC)	Spans	LLM	In / Out / Cached Tokens	Cache Hit %	Status

In / Out / Cached Tokens — display as three values separated by /, e.g. 12 450 / 320 / 8 100. Show — for Cached on a specific row when that trace has cache_unsupported == True. Cache Hit % — cache_read_tokens / (input_tokens - cache_creation_tokens) * 100, formatted as 42 %. Show — for rows where cache_unsupported == True.

If show_cache_cards is False (all traces are Langfuse with no cache data), drop the "Cached Tokens" and "Cache Hit %" columns and use the 5-column layout:

Trace ID	Start Time (UTC)	Spans	LLM	In / Out Tokens	Status

Rules:

• For > 10 traces: omit rows with ✅ OK status — only include failed (❌) and warning (⚠) traces to keep the report compact.
• For ≤ 10 traces: include all traces regardless of status.
• Trace IDs must never be truncated — display full 32-character hex. Use white-space: nowrap; font-family: monospace on the Trace ID cell.

Issues and Warnings tables

#	Span	Type	Description	Evidence	Affected Traces

HTML rendering rules — apply these exactly to prevent column overflow:

1. Table layout — always set table-layout: fixed; width: 100% and declare explicit <colgroup> widths so the browser cannot let any column expand beyond its allocation:

   <table style="table-layout:fixed; width:100%">
     <colgroup>
       <col style="width:3%">   <!-- # -->
       <col style="width:12%">  <!-- Span -->
       <col style="width:7%">   <!-- Type -->
       <col style="width:28%">  <!-- Description -->
       <col style="width:28%">  <!-- Evidence -->
       <col style="width:22%">  <!-- Affected Traces -->
     </colgroup>
     …
   </table>
   

2. Every <td> and <th> — add overflow: hidden to hard-clip any content that would otherwise bleed into the next column:

   <td style="overflow:hidden; word-break:break-word">…</td>
   

3. Evidence column — allow wrapping and use <br> between individual evidence items. Do not set a fixed max-width — let the percentage-based <colgroup> width scale with the container:

   <th>Evidence</th>
   <td style="overflow:hidden; word-break:break-word">
     <code>field: value</code><br>
     <code>field: value</code>
   </td>
   

4. Affected Traces column — use word-break: break-all (not nowrap) so long hex IDs wrap within the cell, and put each ID on its own line:

   <th>Affected Traces</th>
   <td style="overflow:hidden; word-break:break-all; font-family:monospace; font-size:11px">
     tid1<br>tid2<br>…
   </td>
   

For Observations (prose), end each bullet with affected trace IDs in parentheses.

Cache efficiency observations — add the following bullets when applicable. Skip all cache observations entirely when show_cache_cards is False (all-Langfuse batch with no cache data):

• No cache attributes found (OTel only) — the format is OTel and neither gen_ai.usage.cache_read_input_tokens nor llm.token_count.cache_read was present on any LLM span. The provider may not support prompt caching, or the traceloop-sdk version predates cache attribute emission. (list affected trace IDs)
• Zero cache reads on multi-turn thread — cache_read_input_tokens == 0 across all turns in thread <thread_id> (≥ 3 turns). The cache is never warmed; the static system prompt prefix likely changes per turn or is too short to cache. (list affected trace IDs)
• Cache write/read imbalance — total_cache_creation_tokens (N) is more than 5× total_cache_read_tokens (M) across the batch. The prompt-cache prefix is being written but not re-used — likely a cache TTL expiry or a prefix that varies each turn. (list affected trace IDs)

Fix Recommendations

For each critical issue and warning:

Issue: <description>
Root Cause: <likely cause>
Recommended Fix:
- <actionable step 1>
- <actionable step 2>

Cache hit rate fix recommendation template — include when cache hit rate < 20 % or zero cache reads observed on multi-turn threads:

Issue: Low prompt-cache hit rate (<X % batch average)
Root Cause: The system prompt either varies per turn, is not positioned at the start of
            the messages array, or the cache TTL expired between turns in long-gap threads.
Recommended Fix:
- Pin the static system prompt as the first message so the LLM provider can cache the
  longest common prefix across turns.
- Avoid injecting dynamic content (timestamps, user IDs, session state) into the system
  prompt — move that to the user turn instead.
- For Anthropic: add `cache_control: {"type": "ephemeral"}` to the last static content
  block to explicitly mark the cacheable prefix boundary.
- For OpenAI: ensure repeated calls use an identical token-level prefix of ≥ 1 024 tokens;
  caching is automatic but the prefix must be byte-for-byte identical.
- Verify the inter-turn gap is within the provider cache TTL: ~5 min for OpenAI and
  Anthropic ephemeral caches. For long-running sessions, refresh the cache before TTL
  expiry by issuing a short warm-up prompt.
- If cache attributes are absent entirely, upgrade the traceloop-sdk to a version that
  emits `gen_ai.usage.cache_read_input_tokens` / `gen_ai.usage.cache_creation_input_tokens`.

Render as create_html_artifact for 3+ issues; inline for fewer.

HTML report outer wrapper — use width: 80% centred with margin: 0 auto so the report occupies 80 % of the page width:

<div style="width:80%; margin:0 auto; padding:20px 0; font-family:-apple-system,'Segoe UI',system-ui,sans-serif; font-size:14px; line-height:1.6">
  <!-- all report content here -->
</div>

---

Step 6: Offer conversational flow report (multi-trace analysis only)

After delivering the bug report, ask:

"Would you like a conversational flow report for any specific trace? If so, provide a trace ID and I'll pull all traces under the same thread and produce a full thread-level flow report."

Use ask_followup_question with suggestions drawn from the failing trace IDs.

---

Step 7: Produce conversational flow report for a thread

A conversational flow report is thread-scoped — one thread ID may span multiple traces (one per user turn).

7.1 Extract the thread_id

Use detect_format() from Step 3C, then read the thread ID from the correct location:

import json

with open(f"data/{trace_id}.json") as f:
    raw = json.load(f)

fmt = detect_format(raw)
thread_id = None

if fmt == 'otel':
    spans = get_spans(raw)
    for s in spans:
        for a in s.get('attributes', []):
            if a['key'] in ('traceloop.association.properties.thread_id', 'thread.id'):
                thread_id = a['value']['stringValue']
                break
        if thread_id:
            break
else:  # langfuse
    # 1. Check top-level metadata.attributes (most reliable)
    meta_attrs = (raw.get('metadata') or {}).get('attributes', {})
    thread_id = (
        meta_attrs.get('thread.id') or
        meta_attrs.get('thread_id') or
        meta_attrs.get('langfuse.session.id') or
        raw.get('sessionId')   # Langfuse session == wxO thread
    )
    # 2. Fall back to any observation's metadata.attributes
    if not thread_id:
        for obs in raw.get('observations', []):
            obs_attrs = (obs.get('metadata') or {}).get('attributes', {})
            thread_id = obs_attrs.get('thread_id') or obs_attrs.get('thread.id')
            if thread_id:
                break

print(f"format: {fmt}, thread_id: {thread_id}")

7.2 Collect all traces for the thread

First scan the local cache for matching thread_id, then search the API for any traces not yet downloaded. Re-run the local scan after downloading new files. If the wxO token is expired, work with cached traces only and note how many turns were found vs. potentially missing.

7.3 Build the thread timeline

Sort all matched traces by start_utc ascending. For each trace call detect_format() + normalize_span() and extract using the unified field names:

What to extract	OTel span name	Langfuse observation name	Field on normalized span
User message	POST /orchestrate/runs or LangGraph.workflow	POST /chat/completions GENERATION	input (top-level content string)
LLM call count	count of WatsonxChatModel.chat spans	count of WatsonxChatModel.chat GENERATIONs	name == 'WatsonxChatModel.chat'
Input / output tokens	llm.token_count.prompt / .completion attrs	llm.usage.prompt_tokens / .completion_tokens attrs (auto-added)	get_attr(attrs, 'llm.usage.prompt_tokens') etc.
Cache tokens	gen_ai.usage.cache_read_input_tokens attr	usage.cacheReadInputTokens (raw obs fallback)	get_cache_tokens(nspan, raw_obs)
Collaborator routing	chat_with_collaborator_*.tool span output	SPAN obs chat_with_collaborator_* → output.content	output field
Collaborator steps	collaborator.task, tools.task, entrypoint.task spans	CHAIN obs collaborator, tools, LangGraph[collaborator:*]	name, duration_ms, output
Final answer	answer.task span output	CHAIN obs answer → output	output field
Error flag	status_error == True on any span	status_error == True on any obs	n['status_error']

7.4 Render the conversational flow report

Produce a create_html_artifact with four sections:

1. Thread Metadata — thread ID, turn count, agent, environment, start/end time, total duration, overall outcome.
2. Conversation Turns — vertical timeline per turn: user message, routing decision, collaborator steps, response delivered (or red "NO RESPONSE" flag), duration, token counts (in / out / cached), cache hit % for the turn, error flags.
3. Span Timing Breakdown — table of every span across all turns: Turn | Seq | Span name | Actor | Duration (ms) | Status.
4. Root Cause Summary — two-column table comparing what worked vs. what failed.

---

Step 8: Offer further follow-up actions

After the conversational flow report, offer:

1. Deep-dive a specific span — re-read the raw data/{tid}.json, detect format, and show full input / output content (traceloop.entity.input / traceloop.entity.output for OTel; input / output top-level fields for Langfuse). Use normalize_span() so the same display code works for both.
2. Re-run after a fix — guide the user to re-export and re-inspect after applying a fix.
3. Batch analysis — loop over multiple raw trace files directly using the inline Python pattern from Step 4.
4. Compare two traces — diff the span sequences (names, durations, collaborator calls) of two raw traces to identify regressions.