ax-agent-observability

name	ax-agent-observability
description	This skill helps an LLM generate correct AxAgent observability code using @ax-llm/ax. Use when the user asks about axGlobals.onUsage, usageContext, centralized or multi-tenant usage accounting, actorTurnCallback, onContextEvent, agentStatusCallback, onFunctionCall, reportSuccess, reportFailure, getChatLog(), getUsage(), resetUsage(), debug traces, progress updates, or telemetry for AxAgent runs.
version	23.0.14

AxAgent Observability Rules (@ax-llm/ax)

Use this skill when an agent needs runtime visibility, progress reporting, tracing, usage accounting, or chat-log access. For ordinary agent setup use ax-agent. For RLM runtime policy use ax-agent-rlm. For memories and dynamic skill loading use ax-agent-memory-skills.

Choose The Smallest Hook

• Need a quick prompt/runtime trace during development -> start with debug: true.
• Need structured per-turn code, raw runtime result, formatted output, provider thoughts, or actor stage -> use actorTurnCallback.
• Need context-pressure and compaction telemetry -> use onContextEvent.
• Need real-time task progress emitted by actor code -> use agentStatusCallback.
• Need every runtime function call before execution -> use onFunctionCall.
• Need model prompts/responses after a run -> use getChatLog().
• Need centralized chat/embed usage across APIs, users, agents, and services -> use axGlobals.onUsage plus usageContext.
• Need token usage by actor/responder -> use getUsage() and resetUsage().
• Need usage split by context and task stages -> use getStagedUsage().
• Need Ax program traces -> use getTraces().
• Do not add multiple hooks unless the user clearly needs each output stream.

Global Runtime Defaults

OpenTelemetry and debug defaults come from the shared Ax runtime surface:

import { axGlobals, axCreateDefaultColorLogger } from '@ax-llm/ax';
import { trace } from '@opentelemetry/api';

axGlobals.tracer = trace.getTracer('agent-app');
axGlobals.debug = true;
axGlobals.logger = axCreateDefaultColorLogger();
axGlobals.onUsage = (event) => usageQueue.enqueue(event);

These globals are live defaults for future AI, AxGen, AxFlow, and agent-internal model calls. Per-call or explicitly configured options still override axGlobals. Use AxAgent callbacks below when the caller needs structured agent-turn events rather than OpenTelemetry spans or debug logs.

Centralized Usage Observer

Use the process-wide usage observer for application accounting across many agents, API routes, tenants, and users. Keep getUsage() for inspecting one agent instance after a run.

import { axGlobals } from '@ax-llm/ax';

axGlobals.onUsage = (event) => {
  usageQueue.enqueue(event); // Must return immediately.
};

await supportAgent.forward(
  llm,
  { query: request.body.query },
  {
    usageContext: {
      tenantId: auth.tenantId,
      userId: auth.userId,
      requestId: request.id,
      runId: crypto.randomUUID(),
      feature: 'support-chat',
      attributes: { environment: 'production' },
    },
  }
);

Rules:

• The observer receives one immutable normalized event for each completed chat or embedding call that reports provider usage. A fully consumed stream emits once; an unconsumed or cancelled stream may not emit.
• Events include operation, ai, model, normalized tokens, streaming, optional context, and available session or remote request IDs.
• Put stable attribution such as application or environment in AI service-level usageContext. Put tenant, user, request, run, and feature attribution in per-call or per-forward usageContext.
• Per-call context overrides service defaults. attributes are shallow-merged.
• The observer is best-effort and fail-open. Ax does not await it and ignores observer failures, so synchronously enqueue and persist or aggregate out of band.
• The registration is process-wide. A new assignment replaces the previous observer; set axGlobals.onUsage = undefined during test teardown or shutdown when appropriate.
• In multi-process or serverless deployments, send events to a shared durable pipeline. Do not treat an in-memory total as application-wide accounting.
• Keep identifiers opaque and attributes low-cardinality. Avoid prompts, responses, secrets, and other sensitive payloads.
• Token events do not estimate currency cost. Apply a versioned provider/model pricing table downstream.

For direct AI calls and the complete event shape, also read ax-ai.

Actor Turn Callback

Use actorTurnCallback when the caller needs structured telemetry for each actor turn.

What it gives you:

• code: the normalized JavaScript code the actor produced
• stage: which actor produced the turn (distiller or executor)
• result: the raw untruncated runtime return value from executing that code
• output: the formatted action-log output string after Ax normalizes and truncates it for prompt replay
• thought: the actor model's thought field when showThoughts is enabled and the provider returns one
• executorResult: the full actor payload returned by the current actor stage, kept under this historical field name for compatibility
• isError: whether the execution path for that turn was treated as an error
• usage: token usage for this actor turn only
• model: model used for this turn when explicitly set through executorModelPolicy
• chatLogMessages: raw ChatML conversation for this turn, populated only when an actor turn callback is set

Use it for:

• debug UIs that want to show code plus raw runtime results
• tracing and analytics
• capturing thought for internal diagnostics when supported by the provider
• storing per-turn execution artifacts without scraping the prompt/action log

Important:

• output is not raw stdout; it is the formatted replay string used in the action log.
• result is the raw runtime result before Ax applies type-aware serialization and budget-proportional truncation.
• thought is optional and only appears when the underlying AxGen call had showThoughts enabled and the provider actually returned a thought field.
• actionLogEntryCount and guidanceLogEntryCount reflect the live log sizes after the turn is processed, including resumed runs.
• actorTurnCallback fires for the configured agent instance. Child agents passed through functions: [...] should define their own callback if you need their internal actor turns; use onFunctionCall on the parent to observe the parent-side child-agent invocation.

Good pattern:

const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  actorTurnCallback: ({
    stage,
    turn,
    actionLogEntryCount,
    guidanceLogEntryCount,
    code,
    result,
    output,
    thought,
    isError,
    usage,
    model,
  }) => {
    console.log({
      turn,
      stage,
      model,
      actionLogEntryCount,
      guidanceLogEntryCount,
      isError,
      code,
      rawResult: result,
      replayOutput: output,
      thought,
      usage,
    });
  },
  executorOptions: {
    model: 'gpt-5.4-mini',
    showThoughts: true,
  },
});

Callback type:

actorTurnCallback?: (turn: {
  stage: 'distiller' | 'executor';
  turn: number;
  actionLogEntryCount: number;
  guidanceLogEntryCount: number;
  executorResult: Record<string, unknown>;
  code: string;
  result: unknown;
  output: string;
  isError: boolean;
  thought?: string;
  usage?: AxProgramUsage[];
  model?: string;
  chatLogMessages?: ReadonlyArray<{ role: string; content: string }>;
}) => void | Promise<void>;

actorTurnCallback?: (turn: {
  stage: 'distiller' | 'executor';
  turn: number;
  actionLogEntryCount: number;
  guidanceLogEntryCount: number;
  executorResult: Record<string, unknown>;
  code: string;
  result: unknown;
  output: string;
  isError: boolean;
  thought?: string;
  usage?: AxProgramUsage[];
  model?: string;
  chatLogMessages?: ReadonlyArray<{ role: string; content: string }>;
}) => void | Promise<void>; // deprecated alias

Context Event Observability

Use onContextEvent when the caller needs structured telemetry about prompt pressure and compaction. It does not change model behavior directly; it is for logs, evals, and dashboards.

Events:

• budget_check: character-based prompt pressure before an actor turn, with detailed metrics kept out of the actor prompt
• checkpoint_created / checkpoint_cleared: checkpoint lifecycle events with covered turns and reason
• tombstone_created: compact resolved-error summary creation
• relevance_ranking: emitted once per ranked domain per forward when relevanceRanking is enabled; carries domain ('modules' | 'skills' | 'memories'), the shortlist ({ id, score }[], most relevant first), and suppressed (true when the low-confidence guard emitted no hint)
• field_auto_promoted: emitted once per field per run when autoUpgrade keeps an oversized undeclared input value runtime-only; carries fieldName, originalChars, and promptPreviewChars (undefined when no inline preview was kept)

To measure whether the advisory hint helps, join per forward: relevance_ranking.shortlist ids against what the actor then loaded — for modules the internal discover calls (onFunctionCall with kind: 'internal', name: 'discover', args.request) plus the module part of external qualifiedNames; for skills onLoadedSkills / used(id); for memories onLoadedMemories / used(id).

Rules:

• contextPressure in the actor prompt is intentionally compact (ok, watch, critical plus one short instruction).
• Budget metrics are character-based for provider neutrality and are exposed through onContextEvent, not the actor prompt.
• Callback errors are swallowed so telemetry cannot break the agent run.
• Do not scrape actor prompts for pressure metrics.

const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  contextPolicy: { preset: 'checkpointed', budget: 'balanced' },
  onContextEvent: (event) => {
    if (event.kind === 'budget_check') {
      console.log(event.pressure, event.mutablePromptChars);
    }
  },
});

Type:

onContextEvent?: (event: AxAgentContextEvent) => void | Promise<void>;

Agent Status Callback

Use agentStatusCallback when the caller wants real-time progress updates from the actor. When set, the actor can call await reportSuccess(message) and await reportFailure(message) in its JavaScript turns.

const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  agentStatusCallback: (message, status) => {
    console.log(`[${status}] ${message}`);
  },
});

Rules:

• agentStatusCallback receives (message: string, status: 'success' | 'failed').
• When set, the actor prompt automatically includes reportSuccess(message) and reportFailure(message) as available runtime functions.
• The actor is instructed to keep the user updated on task progress.
• reportSuccess and reportFailure are reserved runtime names when the callback is configured.
• Child agents inherit the callback via the RLM config.

Type:

agentStatusCallback?: (
  message: string,
  status: 'success' | 'failed'
) => void | Promise<void>;

On Function Call

Use onFunctionCall when the caller wants to observe every function call the actor makes from the JS runtime. It fires before the underlying function runs.

const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  functions: [helperAgent, lookupOrderTool],
  onFunctionCall: ({ name, qualifiedName, args, kind }) => {
    console.log(`[${kind}] ${qualifiedName}`, args);
  },
});

Rules:

• Receives { name, qualifiedName, args, kind }.
• name is the bare function name, e.g. 'lookupOrder'.
• qualifiedName is the namespaced name as the actor sees it, e.g. 'tools.lookupOrder'; for un-namespaced runtime globals it equals name.
• args is the resolved positional/named arguments object (Record<string, unknown>).
• kind is 'external' for caller-registered functions.
• kind is 'internal' for agent-injected globals: child agents, discover, recall, and used.
• Fires once per call, before the function executes.
• Errors thrown inside the callback are swallowed so they cannot break the actor loop.
• This is independent from the DSP-layer onFunctionCall on AxProgramForwardOptions; that hook is for LLM tool-calls and never fires under AxAgent because AxAgent injects functions as runtime globals.

Type:

onFunctionCall?: (call: {
  name: string;
  qualifiedName: string;
  args: Record<string, unknown>;
  kind: 'internal' | 'external';
}) => void | Promise<void>;

Chat Log, Usage, And Traces

AxAgent exposes actor and responder sub-programs. getChatLog() returns the same flat AxChatLogEntry[] shape as AxGen and AxFlow; use each entry's optional name field to distinguish distiller, executor, and responder. getUsage() returns token usage split by actor/responder.

getChatLog(), getUsage(), and trace export refresh the agent-level snapshot from the distiller, executor, and responder before returning. This remains true after forward() throws, so successful model calls made before an actor limit, provider error, cancellation, or responder error are still observable and are not silently dropped from usage accounting.

getChatLog()

Returns the full normalized chat history after any .forward() call. Each entry is one ai.chat() round-trip. Actor stages accumulate one entry per turn; the responder typically has one entry.

const log = myAgent.getChatLog();

for (const entry of log) {
  console.log(entry.name, entry.model);
  for (const msg of entry.messages) {
    console.log(`[${msg.role}]`, msg.content);
  }
}

Each AxChatLogEntry captures the full prompt sent to the model and its response:

type AxChatLogMessage =
  | { role: 'system'; content: string }
  | { role: 'user'; content: string }
  | { role: 'assistant'; content: string }
  | { role: 'tool'; name: string; content: string };

type AxChatLogEntry = {
  name?: string; // e.g. "distiller", "executor", "responder"
  model: string;
  messages: AxChatLogMessage[];
  modelUsage?: AxProgramUsage;
  stage?: 'ctx' | 'task';
};

getUsage()

Returns token usage split by actor/responder. Each sub-array contains one AxProgramUsage entry per model/run, merged by (ai, model) key.

const usage = myAgent.getUsage();
// { actor: AxProgramUsage[], responder: AxProgramUsage[] }

console.log('Actor tokens:', usage.actor[0]?.tokens);
console.log('Responder tokens:', usage.responder[0]?.tokens);

getStagedUsage()

Returns usage split by pipeline stage. The ctx stage has the distiller actor only; the task stage has the executor actor plus responder.

const staged = myAgent.getStagedUsage();
console.log(staged.ctx?.actor);
console.log(staged.task.actor);
console.log(staged.task.responder);

getTraces()

Returns Ax program traces for the agent pipeline. Use it when the caller needs trace data rather than chat messages or token summaries.

const traces = myAgent.getTraces();

resetUsage()

Resets both actor and responder usage at once:

myAgent.resetUsage();

Type signatures:

// AxAgent
agent.getChatLog(): readonly AxChatLogEntry[]
agent.getUsage(): { actor: AxProgramUsage[]; responder: AxProgramUsage[] }
agent.getStagedUsage(): { ctx?: AxAgentUsage; task: AxAgentUsage }
agent.getTraces(): AxProgramTrace[]
agent.resetUsage(): void

// AxGen / AxFlow
gen.getChatLog(): readonly AxChatLogEntry[]
gen.getUsage(): AxProgramUsage[]

Do Not Generate

• Do not add both debug: true and actorTurnCallback unless the user wants both unstructured prompt/runtime visibility and structured telemetry.
• Do not scrape actor prompts or action logs when a callback exposes the data directly.
• Do not let observability callback failures break the agent run; Ax swallows callback errors for telemetry hooks.
• Do not use DSP-layer onFunctionCall when the user wants AxAgent runtime function calls.
• Do not enable showThoughts unless the user needs provider thought diagnostics and the provider supports it.
• Do not use getUsage() as the centralized source of truth across shared agents or processes.
• Do not perform database or network work inline in axGlobals.onUsage; enqueue and return.