maple-agent-tracing-spring-ai

# Maple agent tracing for Spring AI Goal: every conversation with the Spring AI app shows up in Maple **Agent Sessions** as exactly one session, one turn per `ChatClient` call, with transcript, model calls, tool calls (failures marked), sub-agent lanes and tokens. Mechanism: Spring AI's Micrometer Observations → `micrometer-tracing-bridge-otel` → OpenTelemetry SDK → OTLP/HTTP to Maple, all from `spring-boot-starter-opentelemetry`. Out of the box: sampling is 10%, prompts/replies never reach spans (`log-prompt`/`log-completion` only log to SLF4J), and thrown tool errors end the span OK. Steps 2-5 fix all three. ## Step 0: Detect versions and existing setup 1. Read `pom.xml` / `build.gradle(.kts)`: Spring Boot version, `spring-ai-bom` version, model starter (`spring-ai-starter-model-*`). Target Spring AI 2.0.x (verified 2.0.1) on Boot 4.x (verified 4.1.1), Java 17+. Spring AI 1.1.x on Boot 3.5: see Step 2d. 2. Grep for existing tracing: `micrometer-tracing-bridge`, `spring-boot-starter-opentelemetry`, `opentelemetry-exporter-otlp`, `management.otlp`, `management.opentelemetry`, `management.tracing`, `-javaagent`, `opentelemetry-javaagent`, `OTEL_EXPORTER_OTLP`, `ObservationFilter`, `ObservationPredicate`, `ToolExecutionExceptionProcessor`. - Existing Boot tracing to another backend: Boot has one OTLP span exporter. Ask the user whether to repoint it at Maple; to keep both, add a second exporter via an `OtlpHttpSpanExporter` bean only if they insist. - OTel Java agent attached (`-javaagent:...opentelemetry-javaagent.jar`): see Step 2c. - Existing `ToolExecutionExceptionProcessor` bean: patch it (Step 5) instead of adding the one in Step 3. 3. Find every `ChatClient` call site (`.prompt(`, `.call()`, `.stream()`) and where the conversation/thread id lives in the request. Find every `ChatClient.Builder` (each role/sub-agent). ## Step 1: Key and region - US: `https://ingest.maple.dev`. EU: `https://ingest.eu.maple.dev`. - Header: `Authorization=Bearer <key>`. Protocol: OTLP/HTTP protobuf (Boot's default transport). - Key in the user's prompt: use it. No key: use the literal `MAPLE_TEST` (ingest accepts and discards it) and tell the user to replace it with their key from **Settings → Ingestion**. - Private `maple_sk_` keys never go in browser code. - Follow the repo's existing secret/env convention (`${ENV_VAR}` placeholders, profile files, Vault/Config Server). If there is none, inline in `application.properties` is acceptable because ingest keys are write-only. - A key from `${MAPLE_INGEST_KEY}` must never stop the app or send an empty `Bearer ` (opaque 401). Use the empty default `${MAPLE_INGEST_KEY:}` (a bare `${MAPLE_INGEST_KEY}` fails startup when unset) and turn export off in `main` when the key is missing, before `SpringApplication.run` (Step 2b shows it). That check reads the process environment and Boot does not read `.env` files, so export the variable where the JVM starts. ## Step 2: Install and export ### 2a. Dependencies (Boot 4, Spring AI 2.0) Keep the existing `spring-ai-bom` import and model starter. If there is no BOM yet, import `org.springframework.ai:spring-ai-bom:2.0.1` (`<type>pom</type><scope>import</scope>` in `dependencyManagement`; Gradle `implementation(platform("org.springframework.ai:spring-ai-bom:2.0.1"))`). Add: ```xml <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-opentelemetry</artifactId> </dependency> ``` Gradle: `implementation("org.springframework.boot:spring-boot-starter-opentelemetry")`. Actuator is not needed on Boot 4 (verified); keep it if the app already has it. ### 2b. Properties Add to `application.properties` (or the YAML equivalent): ```properties spring.application.name=<service name, e.g. support-agent> management.opentelemetry.tracing.export.otlp.endpoint=https://ingest.maple.dev/v1/traces management.opentelemetry.tracing.export.otlp.headers.Authorization=Bearer ${MAPLE_INGEST_KEY:} management.tracing.sampling.probability=1.0 management.opentelemetry.resource-attributes.deployment.environment.name=<env> management.otlp.metrics.export.url=https://ingest.maple.dev/v1/metrics management.otlp.metrics.export.headers.Authorization=Bearer ${MAPLE_INGEST_KEY:} maple.ai.capture-content=true ``` Then, in the application's `main`, disable export when the key is unset (merge with an existing `main`; keep its `SpringApplication` call): ```java public static void main(String[] args) { // A missing key disables export; it never stops the app. var mapleKey = System.getenv("MAPLE_INGEST_KEY"); if (mapleKey == null || mapleKey.isEmpty()) { System.err.println("MAPLE_INGEST_KEY is not set; Maple telemetry export is disabled"); System.setProperty("management.tracing.export.enabled", "false"); System.setProperty("management.otlp.metrics.export.enabled", "false"); } SpringApplication.run(Application.class, args); } ``` - The endpoint property takes the FULL URL including `/v1/traces` (Boot does not append it). - `management.tracing.sampling.probability=1.0` is REQUIRED. Default is `0.1`: 90% of turns silently missing. - The starter also exports metrics, to `localhost:4318` by default. Either point them at Maple (above) or set `management.otlp.metrics.export.enabled=false`. Never leave the default. - Streamed tokens: Spring AI 2.0's OpenAI model requests usage on streams by default. If the app sets ANY `spring.ai.openai.chat.stream-options.*` property, also set `spring.ai.openai.chat.stream-options.include-usage=true` (once stream options exist, unset means false and streamed turns report no tokens). - Boot 4.1+ also maps `OTEL_EXPORTER_OTLP_ENDPOINT` (base URL, Boot appends `/v1/traces`), `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES`. Use them if the repo configures via env; keep the sampling property. - Do NOT set `management.opentelemetry.tracing.limits.max-attribute-value-length`: truncated JSON content no longer parses and Maple drops it. - Leave `spring.ai.chat.observations.log-prompt/log-completion` and `spring.ai.chat.client.observations.*` as they are. They only log; they never put content on spans. - `@SpringBootTest` disables tracing export; add `@AutoConfigureTracing` only if the user wants spans from tests. ### 2c. OTel Java agent present The agent does not turn Micrometer Observations into spans, so Steps 2a-2b and 3 are still required. But agent + starter as-is is BROKEN: Boot's SDK and the agent don't share context, so every Spring AI span becomes its own trace (no hierarchy, model calls split from their session). Verified with agent 2.31.1. Either remove the agent for this service, or hand Micrometer the agent's `OpenTelemetry` (add ONLY while the agent is attached; without it `GlobalOpenTelemetry.get()` is a no-op and nothing is traced): ```java @Bean io.opentelemetry.api.OpenTelemetry openTelemetry() { return io.opentelemetry.api.GlobalOpenTelemetry.get(); } ``` Then the agent exports everything: set `OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.maple.dev`, `OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <key>`, `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=<env>` for the agent. Boot's `management.opentelemetry.*` export and sampling properties no longer apply (agent default sampler records everything). Also add `-Dotel.instrumentation.openai-java.enabled=false` as a precaution (no duplicate `chat` spans seen with Spring AI 2.0.1, but the agent ships OpenAI SDK instrumentation). HTTP client spans appear twice (Boot + agent); they are not AI spans and don't affect sessions. ### 2d. Spring AI 1.1 on Boot 3.5 - Deps: `io.micrometer:micrometer-tracing-bridge-otel` + `io.opentelemetry:opentelemetry-exporter-otlp` + `spring-boot-starter-actuator` (required on Boot 3.5; no `spring-boot-starter-opentelemetry`). - Properties: `management.otlp.tracing.endpoint=https://ingest.maple.dev/v1/traces`, `management.otlp.tracing.headers.Authorization=Bearer ...`; sampling property unchanged. Stream usage: `spring.ai.openai.chat.options.stream-usage=true`. - Step 3 class: replace `tools.jackson.databind.json.JsonMapper.shared().writeValueAsString(...)` with a Jackson 2 `com.fasterxml.jackson.databind.ObjectMapper` (wrap the checked `JsonProcessingException`), and delete the two `getToolCallId()` lines (not in 1.1). - OpenAI model property: `spring.ai.openai.chat.options.model` on 1.1 (2.0: `spring.ai.openai.chat.model`). - Boot 4 still accepts the Boot 3 `management.otlp.tracing.*` names but marks them deprecated. ## Step 3: Add the configuration class Create it in a package the app's `@SpringBootApplication` scans (same package or below). Copy verbatim; change only the package: ```java package com.example.agent; import java.util.ArrayList; import java.util.List; import java.util.Map; import io.micrometer.common.KeyValue; import io.micrometer.observation.Observation; import io.micrometer.observation.ObservationFilter; import io.micrometer.observation.ObservationRegistry; import tools.jackson.databind.json.JsonMapper; import org.springframework.ai.chat.client.observation.ChatClientObservationContext; import org.springframework.ai.chat.messages.AssistantMessage; import org.springframework.ai.chat.messages.Message; import org.springframework.ai.chat.messages.ToolResponseMessage; import org.springframework.ai.chat.observation.ChatModelObservationContext; import org.springframework.ai.tool.execution.DefaultToolExecutionExceptionProcessor; import org.springframework.ai.tool.execution.ToolExecutionExceptionProcessor; import org.springframework.ai.tool.observation.ToolCallingObservationContext; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; /** Adds the gen_ai.* attributes Maple reads on top of Spring AI's own observations. */ @Configuration(proxyBeanMethods = false) public class MapleAiObservationConfig { /** Agent name for a ChatClient: `.defaultAdvisors(a -> a.param(AGENT_NAME, "support_agent"))`. */ public static final String AGENT_NAME = "gen_ai.agent.name"; @Bean ObservationFilter mapleGenAiAttributes(@Value("${maple.ai.capture-content:false}") boolean captureContent) { return context -> { if (context instanceof ChatClientObservationContext client) { // One ChatClient call is one agent turn; Spring AI labels it "framework". client.addLowCardinalityKeyValue(KeyValue.of("gen_ai.operation.name", "invoke_agent")); if (client.getRequest().context().get(AGENT_NAME) instanceof String agent) { client.addLowCardinalityKeyValue(KeyValue.of("gen_ai.agent.name", agent)); } } else if (context instanceof ToolCallingObservationContext tool) { tool.addLowCardinalityKeyValue(KeyValue.of("gen_ai.tool.name", tool.getToolDefinition().name())); if (tool.getToolCallId() != null) { tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.id", tool.getToolCallId())); } if (captureContent) { tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.arguments", tool.getToolCallArguments())); if (tool.getToolCallResult() != null) { tool.addHighCardinalityKeyValue(KeyValue.of("gen_ai.tool.call.result", tool.getToolCallResult())); } } } else if (captureContent && context instanceof ChatModelObservationContext chat) { chat.addHighCardinalityKeyValue(KeyValue.of("gen_ai.input.messages", JsonMapper.shared().writeValueAsString(chat.getRequest().getInstructions().stream().map(MapleAiObservationConfig::message).toList()))); if (chat.getResponse() != null) { chat.addHighCardinalityKeyValue(KeyValue.of("gen_ai.output.messages", JsonMapper.shared().writeValueAsString(chat.getResponse().getResults().stream().map(g -> message(g.getOutput())).toList()))); } } return context; }; } // Spring AI hands a failing tool's message back to the model and ends the tool span OK. // Mark the span failed first, then keep the default behaviour. @Bean ToolExecutionExceptionProcessor toolExecutionExceptionProcessor(ObservationRegistry registry) { ToolExecutionExceptionProcessor fallback = DefaultToolExecutionExceptionProcessor.builder().build(); return exception -> { Observation toolCall = registry.getCurrentObservation(); if (toolCall != null) { toolCall.error(exception); } return fallback.process(exception); }; } /** One message in the OpenTelemetry GenAI shape: {role, parts: [...]}. */ private static Map<String, Object> message(Message message) { List<Map<String, Object>> parts = new ArrayList<>(); if (message instanceof ToolResponseMessage toolResponse) { toolResponse.getResponses().forEach(r -> parts.add( Map.of("type", "tool_call_response", "id", r.id(), "response", r.responseData()))); } else { if (message.getText() != null && !message.getText().isEmpty()) { parts.add(Map.of("type", "text", "content", message.getText())); } if (message instanceof AssistantMessage assistant) { assistant.getToolCalls().forEach(call -> parts.add( Map.of("type", "tool_call", "id", call.id(), "name", call.name(), "arguments", call.arguments()))); } } return Map.of("role", message.getMessageType().getValue(), "parts", parts); } } ``` Boot applies `ObservationFilter` beans to the registry automatically; nothing else to register. `spring.ai.tools.observations.include-content` writes `spring.ai.tool.call.arguments/result`, which Maple does not read; the filter's `gen_ai.tool.call.*` keys are the ones read. Content notes: every `chat` span carries the whole conversation so far, so spans grow with long chats (don't cap them; see 2b). With `maple.ai.capture-content=false` no message/tool content leaves the process, except a failed tool's exception message and stack trace (`toolCall.error(exception)`, Step 3); if those can hold user data, pass `error()` an exception without them. To redact content instead, mask values inside `message(...)`. The conversation id and agent names are sent regardless: keep personal data out of them. Kotlin project: translate one to one (e.g. `ObservationFilter { context -> ...; context }`), same beans, same keys.