maple-agent-tracing-opentelemetry

# Maple agent tracing: OpenTelemetry GenAI conventions (any language) Goal: every conversation = one Maple Agent Session. Each user message = one trace rooted at an `invoke_agent` span, with a `chat` span per model call (model, tokens, transcript) and an `execute_tool` span per tool call (args, result, failures). Mechanism: you write the spans. Maple classifies a span only by `gen_ai.operation.name`, groups a trace by `gen_ai.conversation.id`, and reads content only from span attributes. Hand-written spans show as framework "Unidentified" (vendor `unknown:genai`); that is expected. ## Step 0: Detect 1. Language and entry points (web server, workers, scripts, serverless handlers). 2. Is a supported framework the real agent runtime? (`@mastra/core`, `ai`, `agents`/`@cloudflare/ai-chat`, `genkit`/`@genkit-ai/*`, `@openai/agents`/`openai-agents`, `langchain`/`langgraph`, `pydantic-ai`, `crewai`, `google-adk`, `llama-index`, `strands-agents`, `smolagents`, `agno`, `dspy`, `haystack-ai`, `agent-framework`, Spring AI, `litellm`, Claude Agent SDK). If yes, stop and use `maple-agent-tracing-<framework>` instead; use this skill only for the parts that framework doesn't cover, or for the `maple_ai.session.id` wrapper (Step 4). 3. Existing OTel setup. Search for `TracerProvider`, `NodeTracerProvider`, `NodeSDK`, `registerOTel`, `set_tracer_provider`, `opentelemetry-instrument`, `logfire.configure`, `sentry_sdk.init`/`Sentry.init`, `otel.SetTracerProvider`. Exists → add Maple's exporter/processor to it; never create a second provider. 4. Existing GenAI auto-instrumentation on the model client (`@opentelemetry/instrumentation-openai`, `opentelemetry-instrumentation-openai-v2`, OpenLLMetry `Traceloop.init`, OpenInference `OpenAIInstrumentor`, `logfire.instrument_openai`). Pick one source of `chat` spans: either keep that instrumentation (then see `maple-agent-tracing-provider-sdks`) or remove it and write `chat` spans here. Both = every model call twice. 5. Find in the code: the agent loop (where one user message is handled), every model call site, every tool dispatch, sub-agent calls, and where the conversation/chat/thread id lives in the request. ## Step 1: Key and region - US: `https://ingest.maple.dev`. EU: `https://ingest.eu.maple.dev`. - Header: `Authorization=Bearer <key>`. - Key given in the prompt → use it. - No key → use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion. - Never put a private `maple_sk_` key in browser code. - Follow the repo's secret/env convention if it has one. Otherwise inline is acceptable: ingest keys are write-only. ```bash OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>" OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` The exporters append `/v1/traces`. - The SDK exporters read these env vars when the exporter is constructed. If the app loads `.env` (dotenv, `load_dotenv()`, `--env-file`), load it at the top of the tracing module, before the provider is built; otherwise the exporter silently targets `localhost:4318` with no key. - If the header is built from your own env var and it is unset, log one warning (`MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled`) and skip the Maple exporter so the app runs normally. Never throw or exit over the key, and never let it become `Bearer undefined` (opaque 401) or a bare `KeyError` on import. Or inline the key when the repo has no env convention. - 401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust: keys are region-bound, so it usually belongs to the other region. Try the other endpoint. ## Step 2: Install + init Read the reference for the language and adapt it: - TypeScript/Node: `references/typescript.md` - Python: `references/python.md` - Other languages: the language's OTel SDK with an OTLP/HTTP exporter, following the steps below. Rules: - Init module is imported first in every entry point. Set a real `service.name` and `deployment.environment.name`. - Name the tracer after the app (e.g. `support-agent`). Never `openrouter`, `langsmith`, `litellm`, `haystack`, `ai`, `gen_ai`: Maple fingerprints frameworks by scope name and would treat your spans as that framework's. - Keep the project's loop structure; add spans around its existing calls. Use the reference's complete loop only when there is no loop yet. ## Step 3: The three spans (exact keys) `invoke_agent` (kind INTERNAL, name `invoke_agent <agent>`), around one agent run; for a user turn it is the trace root: - `gen_ai.operation.name`=`invoke_agent`, `gen_ai.agent.name`, `gen_ai.conversation.id` (Step 4) - optional: `gen_ai.input.messages` (the user message), `gen_ai.output.messages` (final answer) `chat` (kind CLIENT, name `chat <model>`), around each model call: - at start: `gen_ai.operation.name`=`chat` (or `generate_content`/`text_completion`), `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.system_instructions`, `gen_ai.input.messages` - at end: `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons` (string array), `gen_ai.output.messages`, usage (Step 7), `gen_ai.response.time_to_first_chunk` (double, SECONDS, streamed calls) `execute_tool` (kind INTERNAL, name `execute_tool <tool>`), around each tool call: - `gen_ai.operation.name`=`execute_tool`, `gen_ai.tool.name` (real name), `gen_ai.tool.call.id` (the model's call id), `gen_ai.tool.type`=`function` - `gen_ai.tool.call.arguments`: JSON string of an object - `gen_ai.tool.call.result`: JSON string of the tool's return value; a string return can be set as its plain text. Message JSON (`input.messages`/`output.messages`): array of `{role, parts}`; parts `{type:"text",content}`, `{type:"tool_call",id,name,arguments:<object>}`, `{type:"tool_call_response",id,response}`, `{type:"reasoning",content}`. Output messages add `finish_reason`. `system_instructions` = array of parts, no role: `[{"type":"text","content":"..."}]`. Always a JSON **string** attribute; plain-text messages don't render. Also read: a message may carry `content` (string or part array) instead of `parts`. `gen_ai.response.model` wins over `gen_ai.request.model` when both are set. Legacy spellings are read as fallbacks (use current names in new code): `gen_ai.system` (→ `gen_ai.provider.name`, renamed in semconv 1.37), `gen_ai.usage.prompt_tokens`/`completion_tokens`, whole-value `gen_ai.prompt`/`gen_ai.completion`, `gen_ai.usage.cache_creation.input_tokens` (→ `cache_write`), `gen_ai.usage.total_cost` (→ `cost`). `provider.name` = the API actually called: `openai`, `anthropic`, `gcp.gemini`, `gcp.vertex_ai`, `aws.bedrock`, `azure.ai.openai`, `mistral_ai`, `groq`, `x_ai`, `deepseek`, or `openrouter` for OpenRouter. ## Step 4: Session id (required) - The id is read only from spans that have `gen_ai.operation.name`; on an unclassified span it is ignored. - Set `gen_ai.conversation.id` on the turn's `invoke_agent` span from the app's conversation/chat/thread id. Same value for every message of a conversation; different across conversations. One classified span per trace is enough; every span in the trace joins. - Never: `uuid4()`/`randomUUID()` per request, the trace id, a module-level constant, a per-process default. No real id (single-shot script) → generate one per conversation, not per message, and reuse it. - Sub-agents in the same trace: no id (they inherit the trace's session) or the same id. Two different ids in one trace → the lexically larger silently wins. - Chat backends: one message list per conversation (keyed by that id, persisted), never one global list. Escape hatch, only when a framework's spans carry a session key Maple ignores for that framework (LiteLLM, Haystack): wrap each turn in your own span with `gen_ai.operation.name`=`invoke_agent`, `gen_ai.agent.name`, and `maple_ai.session.id`=<conversation id>, and run the framework inside it. Rules: - Only on your own wrapper span, never on framework spans (it re-vendors the span to `maple`: framework decoding lost, usage read as inclusive of cache). - Use the same value the framework would use for the session. - Not needed for hand-written spans: use `gen_ai.conversation.id`. - The session then shows framework **Maple**. ```ts // chatId comes from your request; frameworkAgent is the framework's agent await tracer.startActiveSpan( "invoke_agent support", { attributes: { "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": "support", "maple_ai.session.id": chatId, }, }, async (span) => { try { return await frameworkAgent.run(message) // the framework's spans nest under this one } finally { span.end() } }, ) ``` ## Step 5: Content - Content = `gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages` (chat), `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result` (execute_tool). On span attributes only: span events, log records, and indexed keys (`gen_ai.prompt.0.content`, `llm.input_messages.0.*`) are not read. - Do not set `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` / `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT`; if the platform sets one, unset it. Truncated JSON is dropped whole. To cap size, drop the oldest messages whole before serializing. - Replace base64 images/files in history with a placeholder part (ingest rejects requests > 20 MiB with 413). - User wants no content → skip those five attributes (everything else still works; transcript empty). Wants redaction → redact inside `toSemconv`/`to_semconv` before serializing, or an OTel Collector `redaction`/`transform` processor. - Never put API keys or `Authorization` headers in any attribute. ## Step 6: Tools, errors, sub-agents - Tool failure: set status ERROR with the error message as description, set `error.type` (exception class or error code), no `gen_ai.tool.call.result`, then return the error to the model as the tool result so the loop continues. Keep the message specific: Maple's tool pages group failures by it. - Maple counts any span as failed if it has status ERROR, a non-empty `error.type`, or `gen_ai.response.status`=`failed`. - Tools that return `{"error": ...}` instead of raising: mark the span failed the same way when you detect it. - Model call failure: status ERROR + `error.type` (HTTP status or exception class), rethrow. - Agent run failure: same on `invoke_agent`. - Sub-agent: call its loop inside the delegating tool's `execute_tool` span, so `execute_tool ask_x` → `invoke_agent x` → its `chat`/`execute_tool`. Distinct `gen_ai.agent.name` per agent (lanes need it). - Delegation detection: an `execute_tool` span whose only child is an `invoke_agent` span is drawn as a delegation into a lane named after the child's `gen_ai.agent.name`; the tool's arguments/result become the lane's input/output. Two agents with the same name share one lane; an `invoke_agent` span without a name gets no lane. - Parallel tools: start each `execute_tool` span inside the turn's context (Node `Promise.all` keeps it; Python threads need `contextvars.copy_context().run`). ## Step 7: Tokens and cost - Usage on `chat` spans only, never cumulative totals on `invoke_agent`. - Keys: `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens`, `gen_ai.usage.reasoning.output_tokens` (ints). `total_tokens` is not read. - Totals as the spec defines them: `input_tokens` = every prompt token, cache reads and writes included; `output_tokens` = every completion token, reasoning included. OpenAI, OpenRouter and Gemini's `promptTokenCount` already count that way. Anthropic's `input_tokens` excludes cache: send `input_tokens + cache_read_input_tokens + cache_creation_input_tokens`. Gemini's `candidatesTokenCount` excludes thoughts: send `candidatesTokenCount + thoughtsTokenCount`. - Streaming OpenAI-compatible: `stream_options: {include_usage: true}`. OpenAI sends usage in an extra last chunk with empty `choices`; OpenRouter always sends usage + `cost` on the chunk carrying `finish_reason`. Read `chunk.usage` before skipping chunks without choices. - OpenRouter (Claude included): `prompt_tokens` already includes `cached_tokens` → provider `openrouter`, copy as is. Optional: `prompt_tokens_details.cache_write_tokens` → `gen_ai.usage.cache_write.input_tokens`. - Cost: `gen_ai.usage.cost` (double, USD) on `chat` spans. OpenRouter returns `usage.cost` → copy it. Other providers return none → compute only if the project has a price table; otherwise leave it (Maple shows "unpriced"; it never prices tokens). - `gen_ai.response.id` always (dedupes against gateway mirrors such as OpenRouter Broadcast). ## Step 8: Flush - Node script/CLI: `await provider.shutdown()` in `finally`. Serverless: `await provider.forceFlush()` before returning (inside `waitUntil`/`after()` if available). Both reject when an export failed: add `.catch((err) => console.error("telemetry flush failed", err))` so a Maple outage can't crash the app. Long-running server: flush on `SIGTERM`, nothing per request. - Python script: `provider.shutdown()` in `finally`. Lambda: `force_flush()` in `finally`. Notebooks/workers: `force_flush()` per cell/task. ## Step 9: Verify If the app has no scriptable entry point (server, REPL, UI only), write a small driver for this run: one conversation id, 2+ turns, at least one tool call, flush before exit. Run one real conversation: 2+ messages with the same id, one streamed reply if the app streams, one tool call, one failing tool if one exists, one sub-agent call if the app delegates; then a second conversation. Wait ~30 s; Maple → Agent Sessions, filter by service name. Check: - [ ] One session per conversation; session id = the conversation id; second conversation = different session; no `trace:<id>` sessions. - [ ] One turn per user message, labeled with it; transcript shows system instructions, user messages, assistant replies, tool calls and results. - [ ] Spans `invoke_agent <agent>` → `chat <model>` / `execute_tool <tool>`, all in the turn's trace (no orphan roots). - [ ] Every `chat` span: model, provider, response id, input+output tokens (streamed ones too), finish reasons; TTFT in seconds on streamed calls. - [ ] Tool spans: real names, call ids matching the model's tool calls, JSON args and results. - [ ] The failing tool is failed with its message; successful tools and model calls are not. - [ ] Sub-agents: own lane with their `gen_ai.agent.name`, same session. - [ ] Cost present only if spans carry `gen_ai.usage.cost`; else "unpriced". - [ ] No attribute contains an API key, `Bearer `, `sk-or-`, `maple_sk_`. With the Maple MCP: `list_agent_sessions` with `search=<conversation id>` returns one row.