maple-agent-tracing-genkit

# Maple agent tracing: Genkit ## Goal One conversation = one Maple Agent Session, one turn per flow run, with the transcript, every model call (model, tokens) and every tool call (name, args, result, failures). Span tree per flow run: ```text supportChat genkit:metadata:subtype=flow, genkit:isRoot=true -> invoke_agent └── generate genkit:type=util (one nested generate per tool-loop step, left unmapped) ├── googleai/gemini-2.5-flash subtype=model -> chat ├── getWeather subtype=tool -> execute_tool └── generate └── googleai/gemini-2.5-flash subtype=model -> chat ``` Beta agents (`ai.defineAgent()` / `defineCustomAgent` / `definePromptAgent` from `genkit/beta`): root span has subtype `agent` and `genkit:metadata:agent:sessionId` (Genkit's session id), which the processor uses as the conversation id. Under it: `runTurn-<n>` (flowStep), `render` (promptTemplate), `generate`, model, tool spans. One trace per `chat.send()`. Known gaps (tell the user, don't try to fix): cost shows as unpriced; `gen_ai.provider.name` is the Genkit plugin prefix (`googleai`, `vertexai`, `openai`, `anthropic`...) rather than the semconv value (`gcp.gemini`...), which is only a label in Maple; media parts are left out of transcripts; `execute_tool` spans have no `gen_ai.tool.call.id` (Genkit doesn't put the call ref on the tool span; the transcript still pairs calls and results through the ids in the model messages when the model plugin sets `ref`). ## Step 0: Detect - `genkit` version in `package.json` / lockfile: need `>= 1.22` (`disableGenkitOTelInitialization` was added in 1.22). Older: upgrade Genkit first. Node.js >= 20. - Go or Python Genkit: stop; this skill covers TypeScript/JavaScript only. Tell the user. - Existing OpenTelemetry: search for `NodeSDK`, `NodeTracerProvider`, `registerOTel`, `@vercel/otel`, `Sentry.init`, `enableTelemetry(`, `enableFirebaseTelemetry(`, `enableGoogleCloudTelemetry(`, `ENABLE_FIREBASE_MONITORING`. - An SDK/provider already exists: reuse it. Add `GenkitForMaple` and one Maple exporting processor to it. Never start a second SDK. - `enableTelemetry({...})` from `genkit/tracing` with custom processors: remove it. Genkit's `enableTelemetry` builds its own bundled `@opentelemetry/sdk-node` 0.52 / `sdk-trace-base` 1.25, and current (2.x) exporters crash inside it (`Cannot read properties of undefined (reading 'name')` on `instrumentationScope`). Don't pass Maple's exporter there. - `enableFirebaseTelemetry()` / `enableGoogleCloudTelemetry()` / `ENABLE_FIREBASE_MONITORING=true`: `disableGenkitOTelInitialization()` makes these no-ops (they go through `enableTelemetry`). Ask the user whether Google Cloud trace export may stop. If they need both, stop and tell them; don't wire two SDKs. - Find every flow (`ai.defineFlow(`), direct `ai.generate(` / `ai.generateStream(` / `prompt(` call site, beta agents (`defineAgent(`, `.chat(`), and how the app deploys (plain Node server, Express `startFlowServer`/`expressHandler`, Next.js `@genkit-ai/next`, Cloud Functions for Firebase `onCallGenkit`, Cloud Run). Find each conversation's id (chat id, thread id, session row). ## Step 1: Key and region - US endpoint `https://ingest.maple.dev`, EU endpoint `https://ingest.eu.maple.dev`. Header `Authorization=Bearer <key>`. - Key in the user's prompt: use it. No key: use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with their key from Settings → Ingestion. - Private `maple_sk_` keys never go in browser code. Ingest keys are write-only. - Follow the repo's existing secret/env convention (`.env`, Firebase `defineSecret`, Secret Manager). If there is none, inlining the ingest key is acceptable. - A 401 `ingest_unauthorized` / "Invalid ingest key" with a key you trust usually means the key belongs to the other region (keys are region-bound): try the other endpoint. ## Step 2: Install ```bash npm install genkit @opentelemetry/sdk-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-proto ``` Use the repo's package manager. `@opentelemetry/api` arrives as a peer; add it explicitly only if the package manager doesn't install peers. Env (or the repo's equivalent): ```bash OTEL_SERVICE_NAME=support-agent OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <key>" ``` Inlining instead of env: `new OTLPTraceExporter({ url: "https://ingest.maple.dev/v1/traces", headers: { authorization: "Bearer <key>" } })` (the full `/v1/traces` path is needed when passing `url`). - The app loads `.env` (`dotenv`, `--env-file`): load it at the top of `instrumentation.ts` (`import "dotenv/config"` as its first line) or run with `--env-file`. The exporter reads the `OTEL_*` vars when it is constructed; otherwise it silently targets `localhost:4318` with no key. - Building the header from a variable (`Bearer ${process.env.MAPLE_INGEST_KEY}`): when it is unset, log one warning (`MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled`) and skip the Maple exporter so the app runs normally. Never throw over the key, and never let it become `Bearer undefined` (opaque 401). ## Step 3: The span processor Create `genkit-for-maple.ts` next to the entry point, verbatim: ```ts // genkit-for-maple.ts import type { ReadableSpan, SpanProcessor } from "@opentelemetry/sdk-trace-base" type Part = { text?: string reasoning?: string toolRequest?: { name: string; ref?: string; input?: unknown } toolResponse?: { name: string; ref?: string; output?: unknown } } type Message = { role: string; content: Part[] } // Genkit message parts to OpenTelemetry GenAI parts. Media parts are left out. function toPart({ text, reasoning, toolRequest, toolResponse }: Part) { if (text !== undefined) return { type: "text", content: text } if (reasoning !== undefined) return { type: "reasoning", content: reasoning } if (toolRequest) { return { type: "tool_call", id: toolRequest.ref, name: toolRequest.name, arguments: toolRequest.input } } if (toolResponse) return { type: "tool_call_response", id: toolResponse.ref, response: toolResponse.output } return undefined } const toParts = (content: Part[]) => content.map(toPart).filter((part) => part !== undefined) function toMessages(messages: Message[]) { return messages.map((m) => ({ role: m.role === "model" ? "assistant" : m.role, parts: toParts(m.content) })) } /** Adds the gen_ai.* attributes Maple reads to Genkit's flow, model and tool spans. */ export class GenkitForMaple implements SpanProcessor { onStart() {} onEnd(span: ReadableSpan) { const attrs = span.attributes const json = (key: string) => { const value = attrs[key] return typeof value === "string" ? JSON.parse(value) : undefined } const name = String(attrs["genkit:name"]) switch (attrs["genkit:metadata:subtype"]) { case "flow": case "agent": { Object.assign(attrs, { "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": name, }) // Set in your flow, or by Genkit for defineAgent() chats const conversationId = attrs["genkit:metadata:conversationId"] ?? attrs["genkit:metadata:agent:sessionId"] if (conversationId !== undefined) attrs["gen_ai.conversation.id"] = conversationId break } case "model": { const input = json("genkit:input") const output = json("genkit:output") const [provider, ...model] = name.split("/") const messages: Message[] = input?.messages ?? [] const system = messages.filter((m) => m.role === "system").flatMap((m) => toParts(m.content)) Object.assign(attrs, { "gen_ai.operation.name": "chat", "gen_ai.provider.name": provider, "gen_ai.request.model": model.join("/") || name, "gen_ai.input.messages": JSON.stringify(toMessages(messages.filter((m) => m.role !== "system"))), }) if (system.length > 0) attrs["gen_ai.system_instructions"] = JSON.stringify(system) if (output?.message) { attrs["gen_ai.output.messages"] = JSON.stringify( toMessages([output.message]).map((m) => ({ ...m, finish_reason: output.finishReason })), ) } if (output?.finishReason) attrs["gen_ai.response.finish_reasons"] = [output.finishReason] if (output?.usage?.inputTokens !== undefined) attrs["gen_ai.usage.input_tokens"] = output.usage.inputTokens if (output?.usage?.outputTokens !== undefined) attrs["gen_ai.usage.output_tokens"] = output.usage.outputTokens break } case "tool": { Object.assign(attrs, { "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": name, "gen_ai.tool.call.arguments": attrs["genkit:input"] ?? "{}", }) const result = json("genkit:output") if (result !== undefined) { attrs["gen_ai.tool.call.result"] = typeof result === "string" ? result : JSON.stringify(result) } break } } } forceFlush() { return Promise.resolve() } shutdown() { return Promise.resolve() } } ``` It maps: | Genkit span (`genkit:metadata:subtype`) | Adds | | --- | --- | | `flow`, `agent` | `gen_ai.operation.name=invoke_agent`, `gen_ai.agent.name=<genkit:name>`, `gen_ai.conversation.id` from `genkit:metadata:conversationId` or `genkit:metadata:agent:sessionId` | | `model` | `chat`, `gen_ai.provider.name` (prefix before `/`), `gen_ai.request.model` (rest), `gen_ai.input.messages` / `gen_ai.system_instructions` from `genkit:input.messages`, `gen_ai.output.messages` + `gen_ai.response.finish_reasons` from `genkit:output`, `gen_ai.usage.input_tokens` / `output_tokens` from `genkit:output.usage` | | `tool` | `execute_tool`, `gen_ai.tool.name`, `gen_ai.tool.call.arguments` (= `genkit:input`), `gen_ai.tool.call.result` (= `genkit:output`, string results as plain text) | Message conversion: role `model` → `assistant`; parts `{text}` → `text`, `{reasoning}` → `reasoning`, `{toolRequest:{name,ref,input}}` → `tool_call`, `{toolResponse:{name,ref,output}}` → `tool_call_response`; `media`, `data`, `custom` parts are dropped. Notes: - It mutates `span.attributes` in `onEnd`. That works because the exporting processor reads the same object; list `GenkitForMaple` BEFORE the exporting processor (required with `SimpleSpanProcessor`, which exports inside `onEnd`). - Optional, only if the user asks for cache/reasoning token detail: `output.usage.cachedContentTokens` → `gen_ai.usage.cache_read.input_tokens`, `output.usage.thoughtsTokens` → `gen_ai.usage.reasoning.output_tokens`. Gemini models (`googleai/`, `vertexai/`) report `outputTokens` without thoughts: for them set `output_tokens` to `outputTokens + thoughtsTokens`. Keep `input_tokens` as Genkit reports it. - Optional provider label mapping (`googleai` → `gcp.gemini`, `vertexai` → `gcp.vertex_ai`): cosmetic, skip unless asked. - Leave `genkit:*` attributes in place; they are what the Genkit Developer UI and Google Cloud views read. - Typecheck the file with the repo's `tsc` (it passes `strict`). ## Step 4: Start OpenTelemetry Create `instrumentation.ts`: ```ts import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto" import { NodeSDK } from "@opentelemetry/sdk-node" import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base" import { disableGenkitOTelInitialization } from "genkit/tracing" import { GenkitForMaple } from "./genkit-for-maple" export const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter()) export const sdk = new NodeSDK({ spanProcessors: [new GenkitForMaple(), spanProcessor] }) if (process.env.GENKIT_ENV !== "dev") { disableGenkitOTelInitialization() sdk.start() } ``` - `import "./instrumentation"` as the FIRST line of every entry point (server, worker, CLI, Cloud Function index). It must run before Genkit's first span: otherwise Genkit's own SDK registers the global tracer provider first and Maple's never receives spans. Adjust the import extension/style to the repo (ESM `.js` suffix, `tsx`, bundler). - Why `disableGenkitOTelInitialization()`: without it Genkit lazily starts a second, bundled NodeSDK on its first span. With Maple's SDK started first that second one mostly loses the global registration race, but it's two SDKs; disable it. - `GENKIT_ENV=dev` is set by `genkit start` (Developer UI). Genkit's own SDK then ships traces to the Developer UI's telemetry server; Maple's SDK can't feed it (Genkit's `TraceServerExporter` reads 1.x span fields). So in dev: Developer UI traces, nothing to Maple. If the user wants Maple in dev too, drop the `if` and tell them the Developer UI trace view goes empty. - Existing provider/SDK: add `new GenkitForMaple()` and the Maple `BatchSpanProcessor` to its span processors (before any exporting processor for GenkitForMaple), keep `disableGenkitOTelInitialization()`, don't create `NodeSDK`.