maple-agent-tracing-google-adk

# Maple agent tracing: Google ADK (Python and TypeScript) Goal: one conversation = one ADK session id = one Maple Agent Session, with the transcript (user, assistant, tool calls and results), model calls, tool calls with arguments/results, failures, and tokens. ADK emits its own OTel spans (scope `gcp.vertex.agent`): `invocation` > `invoke_agent {agent}` > `call_llm` > `generate_content {model}`, plus `execute_tool {tool}`. No instrumentation package is needed. You add: a tracer provider (Runner apps only), the env vars below, one plugin, one span processor. **TypeScript (`@google/adk` in `package.json`): follow [references/typescript.md](references/typescript.md) instead of Steps 0-7 below.** Step 1 (key and region) applies to both. ADK for TypeScript records content differently (Gemini-shaped JSON on `gcp.vertex.agent.*` attributes, no `generate_content` span), so the Python env switches and plugin do not apply there. ## Step 0: Detect versions and existing setup - Read `pyproject.toml` / `requirements*.txt` / `uv.lock`. Require `google-adk>=2.10`; upgrade if lower (before 2.7, `gen_ai.system` is always `gemini` and failed tools are not marked ERROR). - ADK caps `opentelemetry-sdk<=1.42.1`. Never pin a newer OTel package; let the resolver choose. - Find how ADK runs: - `adk web` / `adk api_server` (CLI): ADK builds the tracer provider from `OTEL_EXPORTER_OTLP_*` env. Do NOT register another provider. - `Runner(...)` in the app's own code (FastAPI, worker, script, notebook): nothing is exported unless the app registers a provider. You must add one. - Grep for an existing OTel setup: `set_tracer_provider`, `TracerProvider(`, `logfire.configure`, `sentry_sdk.init`, `maybe_set_otel_providers`, `opentelemetry-instrument`. If one exists, add the Maple processor to that provider instead of creating a second one. - Grep for double instrumentation and remove it if it only exists for tracing: `litellm.callbacks` containing `"otel"`, `openinference-instrumentation-google-adk` / `GoogleADKInstrumentor`, `openinference-instrumentation-litellm`, `openinference-instrumentation-openai`, `opentelemetry-instrumentation-openai-v2`. Ask the user before removing something another backend relies on. ## Step 1: Key and region - US: `https://ingest.maple.dev`. EU: `https://ingest.eu.maple.dev`. - Header: `Authorization=Bearer <key>`. - Key given in the prompt: use it. No key: use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with a key from Settings → Ingestion. - Never put a private `maple_sk_` key in browser code. - Follow the repo's secret/env convention (`.env`, settings module, deployment env). If there is none, inline values are acceptable: ingest keys are write-only. - Key read from env in code: when it is unset, log one warning (`MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled`) and skip the Maple exporter so the app runs normally. Never raise or exit over the key; no bare `KeyError` on import, no header without a key. ## Step 2: Install and initialize ```bash pip install "google-adk>=2.10" opentelemetry-exporter-otlp-proto-http ``` Add `litellm` only if the app uses `google.adk.models.lite_llm.LiteLlm`. Use the repo's package manager (`uv add`, `poetry add`). Env vars (all runtimes, including `adk web`/`api_server`): ```bash OTEL_SERVICE_NAME=<service-name> OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>" OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false ``` - `OTEL_EXPORTER_OTLP_ENDPOINT` gets `/v1/traces` appended. If you use `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` instead, give the full `https://ingest.maple.dev/v1/traces`. - The env vars must be in the process environment when `telemetry.py` builds the exporter. Otherwise it silently targets `localhost:4318` with no key. For Runner apps, create `telemetry.py` next to the entry point: ```py # telemetry.py import json from google.adk.plugins.base_plugin import BasePlugin from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor class SkipDuplicateToolSpans(BatchSpanProcessor): """Drops two ADK tool spans that would count a call twice: the `execute_tool (merged)` summary of parallel calls, and the span of a call paused for confirmation (it runs again, in its own span, once approved).""" def on_end(self, span): if span.name != "execute_tool (merged)" and not span.attributes.get("adk.awaiting_confirmation"): super().on_end(span) class ToolCallAttributes(BasePlugin): """Records each tool call's arguments and result on its `execute_tool` span, and marks the span of a call that is waiting for confirmation.""" def __init__(self): super().__init__(name="tool_call_attributes") async def before_tool_callback(self, *, tool, tool_args, tool_context): trace.get_current_span().set_attribute("gen_ai.tool.call.arguments", json.dumps(tool_args, default=str)) async def after_tool_callback(self, *, tool, tool_args, tool_context, result): span = trace.get_current_span() if tool_context.actions.requested_tool_confirmations: span.set_attribute("adk.awaiting_confirmation", True) span.set_attribute("gen_ai.tool.call.result", json.dumps(result, default=str)) # Reads OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS provider = TracerProvider(resource=Resource.create()) provider.add_span_processor(SkipDuplicateToolSpans(OTLPSpanExporter())) trace.set_tracer_provider(provider) ``` - Import `telemetry` before any `google.adk` or agent import; if you use python-dotenv, call `load_dotenv()` on the line before it. - Existing provider found in Step 0: skip `TracerProvider(...)`/`set_tracer_provider`; call `existing_provider.add_span_processor(SkipDuplicateToolSpans(OTLPSpanExporter()))` right after it is created. Keep `ToolCallAttributes`. - Register the plugin on every `Runner`: `Runner(..., plugins=[telemetry.ToolCallAttributes()])`, appending to existing plugins. For `adk web`/`api_server`, add it to the `App(name=..., root_agent=..., plugins=[...])` in the agent module and do not create a provider (only the plugin is needed). The processor can't be added there without replacing ADK's provider, so `execute_tool (merged)` spans and paused-confirmation spans stay; tell the user tool counts can be inflated under the CLI. ## Step 3: Session id - ADK writes `session.id` as `gen_ai.conversation.id` on `invoke_agent` and `generate_content` spans; Maple groups on it. One `run_async()` = one trace = one turn. - Make sure every turn of a conversation calls `runner.run_async(user_id=..., session_id=<conversation id>, new_message=...)` with the SAME id. Use the app's own chat/thread id. `Runner(..., auto_create_session=True)` creates the session under that id on the first turn. `InMemoryRunner` (ADK samples) has no `auto_create_session`: call `await runner.session_service.create_session(app_name=..., user_id=..., session_id=<conversation id>)` once before the first turn; `plugins=` works the same. - Fix code that calls `session_service.create_session(...)` without `session_id` on every request: that mints a new UUID per turn (and the model loses history). - Never use a process-global constant session id for all users. - Prefer `run_async()`; in servers do not use sync `runner.run()`. ## Step 4: Content - The two OTel env vars from Step 2 put `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`, `gen_ai.tool.definitions` on `generate_content` spans as `[{role, parts}]` JSON. This is the only content Maple reads. - `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` means log records only. Use `SPAN_ONLY`. - `ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false` removes ADK's `gcp.vertex.agent.*` duplicate blobs (Maple ignores them; they double payload size). - Per-request alternative: `RunConfig(telemetry=TelemetryConfig(genai_semconv_stability_opt_in="experimental", capture_message_content=ContentCapturingMode.SPAN_ONLY))`, imports from `google.adk.telemetry.context`. - If the user wants no content: omit `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` and delete the two `gen_ai.tool.call.*` `set_attribute` lines from `ToolCallAttributes` (keep the plugin: it still marks paused-confirmation spans). Tell the user the transcript will be empty. ## Step 5: Tools, errors, sub-agents - A tool call is marked failed (ERROR + `error.type`) when the tool raises, or returns a dict with a non-empty `"error"` key (`error.type=TOOL_ERROR`). - Tools returning `{"status": "error", "error_message": ...}` are NOT marked failed. If the repo uses that format, tell the user and offer to change it to `{"error": ...}` (changes what the model sees; confirm first). - A raised tool exception aborts the run. Only if the user wants the agent to continue, add: ```py class ToolErrorsAsResults(BasePlugin): def __init__(self): super().__init__(name="tool_errors_as_results") async def on_tool_error_callback(self, *, tool, tool_args, tool_context, error): return {"error": str(error)} ``` - Human approval (`FunctionTool(func, require_confirmation=True)`): ADK opens an `execute_tool` span for the paused call and another when the approved call runs. `ToolCallAttributes` marks the paused one and `SkipDuplicateToolSpans` drops it, so each call counts once. Send the approval (`FunctionResponse` named `adk_request_confirmation`) with the same `session_id`: it becomes its own trace in the same session. A rejected call is marked failed (`This tool call is rejected.`). - Every agent needs a distinct `name` (becomes `gen_ai.agent.name`, Maple's lanes). - Agents as tools: use `sub_agents=[LlmAgent(..., mode="single_turn")]`. Each call produces `execute_tool {agent}` (counted as a tool call, result = the sub-agent's reply) and a sibling `invoke_agent {agent}` under the parent agent, not nested; several delegations in one model response run in parallel. Replace `AgentTool(agent=...)` if present and the user agrees: `AgentTool` runs the sub-agent in a fresh in-memory session, so its spans carry a second `gen_ai.conversation.id` and Maple may file the turn under the wrong session. - `SequentialAgent` / `ParallelAgent` / `LoopAgent` need no changes. ## Step 6: Flush - Scripts, CLIs, notebooks, jobs, tests: wrap the work in `try/finally` and call `telemetry.provider.force_flush()` then `telemetry.provider.shutdown()`. - Servers: call `provider.shutdown()` in the shutdown hook (FastAPI `lifespan`). - CPU-throttled serverless (Cloud Run default, Cloud Functions, Lambda): call `provider.force_flush()` before returning each response. ## Step 7: Verify Run one conversation of 2-3 turns with the same session id, one turn calling a tool, then flush. If the app has no scriptable entry point (`adk web` agent package, server, UI only), write a small driver for this run: one session id, 2+ turns, a tool call, flush before exit. With a real key, open Maple → Agent Sessions (`https://app.maple.dev/agent-sessions`, EU `app.eu.maple.dev`); data appears within about a minute. Check: - [ ] Exactly one session for the conversation, framework **Google ADK**, one turn per `run_async()`. A second conversation is a separate session. - [ ] Transcript shows user messages, assistant replies, tool calls with arguments and results. - [ ] LLM calls and input/output tokens are non-zero on every turn, including streamed ones. Model = `gen_ai.request.model` (e.g. `openrouter/openai/gpt-4o-mini`). - [ ] Tool calls carry real function names; no tool named `(merged tools)`; an approved confirmation-gated call counts once. - [ ] A failing tool is counted as an error; successful tools are not. - [ ] Sub-agents appear as separate lanes with their own names. - [ ] No duplicated model calls (no second instrumentor). - [ ] Cost shows "unpriced" (expected: ADK records no cost). With the Maple MCP: `list_agent_sessions` with `search=<session id>` returns one row. Without Maple access (or with `MAPLE_TEST`), both must hold; silence alone proves nothing (no spans is silent too): - The run exits with no `Failed to export span batch` / 401 lines on stderr. - A local console exporter shows the spans. Temporarily add `provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))` (`from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter`), run one turn, and confirm `generate_content` spans have `gen_ai.conversation.id`, `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.usage.input_tokens`; `execute_tool` spans have `gen_ai.tool.call.arguments`. Also useful with a real key when exports fail. Remove it afterwards. Tell the user about the known gaps: cost is unpriced; the model is the requested id, not the served one, and there is no `gen_ai.response.id`. ## Reference notes - Scope: ADK for Python here, ADK for TypeScript in references/typescript.md. ADK for Go and Kotlin emit the same span names; their provider setup is not covered. - Expected trace per turn: ```text invocation └─ invoke_agent assistant ├─ call_llm │ └─ generate_content openrouter/openai/gpt-4o-mini ├─ execute_tool get_weather └─ call_llm └─ generate_content openrouter/openai/gpt-4o-mini ``` - Tokens: `generate_content` carries `gen_ai.usage.input_tokens` / `output_tokens`, plus `gen_ai.usage.cache_read.input_tokens` and `gen_ai.usage.reasoning.output_tokens` when reported (cached inside input, thinking inside output). `call_llm` repeats the same usage; Maple counts it on `generate_content` only, so totals and LLM call count are correct. Streamed turns (`StreamingMode.SSE`) report usage; `LiteLlm` requests it via `stream_options.include_usage`. - Cost: LiteLLM computes a cost but it never reaches ADK's spans; sessions are unpriced. - `AgentTool` detail: Maple keeps one `gen_ai.conversation.id` per trace and picks the larger of the two, which is why the turn can move to another session. - The approval `run_async()` is its own turn, labeled with the original request. - With the settings in this skill, `generate_content` spans carry no provider attribute; doesn't affect grouping, tokens or transcript. - 401 from the exporter (`ingest_unauthorized`, "Invalid ingest key"): with a key you trust, it usually belongs to the other region (keys are region-bound): try the other endpoint. ## Do not - Do not drop or filter `call_llm` spans: `generate_content` would lose its parent.