ai-core-middleware

# Middleware > **Dependency note:** This skill builds on ai-core. Read it first for critical rules. ## Setup — Analytics Tracking Middleware ```typescript import { chat, toServerSentEventsResponse } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' import { trackAnalytics, reportError } from './analytics' export async function POST(request: Request) { const { messages, threadId } = await request.json() const stream = chat({ adapter: openaiText('gpt-5.5'), messages, threadId, middleware: [ { onStart: (ctx) => { console.log('Chat started:', ctx.model) }, onFinish: (ctx, info) => { trackAnalytics({ model: ctx.model, tokens: info.usage?.totalTokens }) }, onError: (ctx, info) => { reportError(info.error) }, }, ], }) return toServerSentEventsResponse(stream) } ``` ## Hooks Reference Every hook receives a `ChatMiddlewareContext` as its first argument, which provides `requestId`, `streamId`, `phase`, `iteration`, `chunkIndex`, `callMetadata`, `model`, `provider`, `signal`, `abort()`, `defer()`, and more. `parentRunId` names the run this one continues. `subagentRunId` is set only inside a subagent and names its card. | Hook | When | Second Argument | | -------------------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | | `onConfig` | Once at startup (`init`) + once per iteration (`beforeModel`) + once at a separate-finalization boundary | `ChatMiddlewareConfig` (return partial to merge) | | `onStructuredOutputConfig` | Once at the separate-finalization boundary | `StructuredOutputMiddlewareConfig` (return partial) | | `onStart` | Once after initial `onConfig` | none | | `onIteration` | Start of each agent loop iteration | `IterationInfo` | | `onShouldContinue` | Whether to start another agent-loop iteration (AND with strategy; `false` stops) | `AgentLoopState` | | `onChunk` | Every streamed chunk | `StreamChunk` (return void/chunk/chunk[]/null) | | `onBeforeToolCall` | Before each tool executes | `ToolCallHookContext` (return decision or void) | | `onAfterToolCall` | After each tool executes | `AfterToolCallInfo` | | `onToolPhaseComplete` | After all tool calls in an iteration | `ToolPhaseCompleteInfo` | | `onUsage` | When `RUN_FINISHED` includes usage data | `UsageInfo` | | `onFinish` | Run completed normally | `FinishInfo` | | `onAbort` | Run was aborted | `AbortInfo` | | `onError` | Unhandled error occurred | `ErrorInfo` | Terminal hooks (`onFinish`, `onAbort`, `onError`) are **mutually exclusive** -- exactly one fires per `chat()` invocation. > **Sampling in `onConfig`:** `temperature`, `topP`, and `maxTokens` are **not** > first-class fields on `ChatMiddlewareConfig`. To adjust sampling from > middleware, return a partial that mutates `config.modelOptions` using the > provider's native key (e.g. OpenAI `temperature` / `max_output_tokens`, > Anthropic `max_tokens`, Ollama nested `options.num_predict`). Returning a > top-level `temperature`/`maxTokens` has no effect. ### Phase values `ctx.phase` is one of: | Phase | When | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `'init'` | Initial setup (before the first `onConfig` snapshot is built). | | `'beforeModel'` | Right before each agent-loop adapter call (`onConfig` re-fires here). | | `'modelStream'` | During model streaming chunks within the agent loop. | | `'beforeTools'` | Before tool execution phase. | | `'afterTools'` | After tool execution phase. | | `'structuredOutput'` | During the separate-finalization adapter call (set for all chunks from `adapter.structuredOutputStream` or the synthesized fallback). Does not occur for native-combined output. | **Separate-finalization path** (adapters without native-combined support): - `onStructuredOutputConfig` fires **before** `onConfig` at the structured-output boundary. - `onConfig` re-fires at the same boundary with `ctx.phase === 'structuredOutput'`, receiving the post-`onStructuredOutputConfig` view of the config (minus `outputSchema`). - `onChunk` and `onUsage` fire for every chunk and usage event emitted by the structured-output call, with `ctx.phase === 'structuredOutput'`. - `onIteration` does **not** fire for finalization — it is agent-loop-only. - **Terminal `info` and structured-output:** `info.usage` / `info.finishReason` / `info.content` reflect the **agent loop's** terminal state, NOT the finalization step. Finalization state is intentionally segregated to keep agent-loop semantics clean. For a tools-less `chat({ outputSchema })` run, `info.usage` is `undefined` and `info.finishReason` is `null` (no agent-loop iteration produced `RUN_FINISHED`). To capture finalization tokens, use `onUsage` — it fires for both agent-loop iterations and the final call. For the structured-output result itself, observe the `structured-output.complete` CUSTOM event in `onChunk`. **Native-combined output:** - The schema-constrained JSON is produced by a normal agent-loop iteration. `onStructuredOutputConfig` does not fire, `ctx.phase` remains `'modelStream'`, and `onIteration` fires for that iteration. - `info.content` includes the structured JSON because it is agent-loop text. Middleware observes the `structured-output.complete` event in `onChunk` during the same phase. **Both paths:** - On successful completion, `onFinish` fires once after the structured result completes. Terminal-hook exclusivity still holds. - By `onFinish`, `ctx.messages` includes the completed terminal assistant messages. Native-combined output keeps the structured result on its terminal assistant message. The separate-finalization path can preserve the agent loop's plain-text message followed by a distinct structured-output message. ## onStructuredOutputConfig A dedicated config hook that fires **only** at the separate-finalization boundary. Use it to transform the JSON Schema sent to the provider (inject `$defs`, strip vendor-incompatible keywords) or to apply structured-output- specific config changes that should not affect the agent-loop adapter calls. **Signature:** ```ts import type { ChatMiddlewareContext, StructuredOutputMiddlewareConfig, } from '@tanstack/ai' // Excerpt of the `ChatMiddleware` interface exported by '@tanstack/ai' interface ChatMiddleware { onStructuredOutputConfig?: ( ctx: ChatMiddlewareContext, config: StructuredOutputMiddlewareConfig, ) => | void | null | Partial<StructuredOutputMiddlewareConfig> | Promise<void | null | Partial<StructuredOutputMiddlewareConfig>> } ``` **`StructuredOutputMiddlewareConfig` shape:** ```ts import type { ChatMiddlewareConfig, JSONSchema } from '@tanstack/ai' // As exported by '@tanstack/ai' interface StructuredOutputMiddlewareConfig extends Omit< ChatMiddlewareConfig, 'tools' > { outputSchema: JSONSchema // The JSON Schema being sent to the provider } ``` Note the `Omit<…, 'tools'>`: there is **no `config.tools`** on this hook. The structured-output call is the final, tool-free call, so reading or returning `tools` here is a compile error, not a no-op. Transform tools in `onConfig` instead. **Ordering rule:** - `onStructuredOutputConfig` fires **before** `onConfig` at the structured-output boundary. - `onConfig` re-fires at the same boundary with `ctx.phase === 'structuredOutput'`, receiving the post-`onStructuredOutputConfig` view of the config (minus `outputSchema`). - Use `onConfig` for general-purpose transforms that apply to every adapter call (agent-loop iterations and the final structured-output call). - Use `onStructuredOutputConfig` when you need to transform the JSON Schema or apply structured-output-specific behavior. ## Core Patterns ### Pattern 1: Analytics and Logging Middleware Use `onStart`, `onFinish`, `onUsage`, and `onError` for comprehensive observability. Use `ctx.defer()` for non-blocking async side effects that should not block the stream. ```typescript import { chat, toServerSentEventsResponse, type ChatMiddleware, } from '@tanstack/ai' import { openaiText } from '@tanstack/ai-openai' const analytics: ChatMiddleware = { name: 'analytics', onStart: (ctx) => { console.log(`[${ctx.requestId}] Chat started — model: ${ctx.model}`) }, onUsage: (ctx, usage) => { console.log(`[${ctx.requestId}] Tokens: ${usage.totalTokens}`) }, onFinish: (ctx, info) => { ctx.defer( fetch('/api/analytics', { method: 'POST', body: JSON.stringify({ requestId: ctx.requestId, model: ctx.model, duration: info.duration, tokens: info.usage?.totalTokens, finishReason: info.finishReason, }), }), ) }, onError: (ctx, info) => { ctx.defer( fetch('/api/errors', { method: 'POST', body: JSON.stringify({ requestId: ctx.requestId, error: String(info.error), duration: info.duration, }), }), ) }, } export async function POST(request: Request) { const { messages, threadId } = await request.json() const stream = chat({ adapter: openaiText('gpt-5.5'), messages, threadId, middleware: [analytics], }) return toServerSentEventsResponse(stream) } ``` ### Pattern 2: Tool Interception Middleware Use `onBeforeToolCall` to validate, gate, or transform tool arguments before execution. Use `onAfterToolCall` to log results and timing. The first middleware that returns a non-void decision from `onBeforeToolCall` short-circuits remaining middleware for that call. ```typescript import type { ChatMiddleware } from '@tanstack/ai' const toolGuard: ChatMiddleware = { name: 'tool-guard', onBeforeToolCall: (ctx, hookCtx) => { // Block dangerous tools if (hookCtx.toolName === 'deleteDatabase') { return { type: 'abort', reason: 'Dangerous operation blocked' } } // Enforce default arguments. `hookCtx.args` is `unknown` — the provider // sent it — so narrow before reading it. No `as` casts. if (hookCtx.toolName === 'search') { const args = typeof hookCtx.args === 'object' && hookCtx.args !== null ? hookCtx.args : {} if (!('limit' in args)) { return { type: 'transformArgs', args: { ...args, limit: 10 }, } } } // Return void to continue normally }, onAfterToolCall: (ctx, info) => { if (info.ok) { console.log(`${info.toolName} completed in ${info.duration}ms`) } else { console.error(`${info.toolName} failed:`, info.error) } }, } ``` **`onBeforeToolCall` decision types:** | Decision | Effect |