ax-cpp-ai

# AxAI Providers For C++ This skill helps an agent write C++ code with the generated Ax package `axllm`. Use the generated package API, examples, and manifests; do not import TypeScript-only APIs unless you are editing the TypeScript package. ## When To Use - Create provider clients or normalize provider options. - Choose a named deployment profile separately from the model ID served by that deployment. - Attach renewable per-request credentials for expiring cloud tokens. - Resolve structured-output modes from the selected profile and model. - Choose between model-list routing, ordered failover, and adaptive operational routing. - Route multimodal requests without flattening native images or files when the selected provider supports them. - Use scripted transports for deterministic no-key examples. - Use provider-api examples only when explicit provider credentials are available. ## Package Facts - Language: C++. - Package: `axllm`. - Package API docs: `API.md` and `axir-api.json`. - Capability manifest: `axir-capabilities.json`. - Runnable examples: `examples/`. - Real network support: yes. - Scripted no-key transport support: yes. - Runtime profiles: `javascript-quickjs`, `python-pyodide`. ## Core Pattern ```cpp #include "axllm/axllm.hpp" auto llm = axllm::ai("openai", { {"apiKey", std::getenv("OPENAI_API_KEY")} }); ``` ## Decision evidence Add (evidence) to boolean/class/score signature outputs to retain probabilities, values, provenance, and native class/score confidence through AxGen. Boolean threshold 0.7 is local to that field; scalar fields still use provider trueThreshold. A plain score returns a continuous zero-based rubric index. Native selected classes and continuous scores are preserved, even when they differ from argmax or a weighted mean. Generative providers elicit probabilities, derive values locally, omit confidence, and use provenance.source=elicited. Ax attaches actual provider/model provenance. The result shape does not make confidence or calibration comparable across models. By default a decision refusal raises a terminal AxDecisionRefusalError without correction retries. Use (evidence, refusal return) for an answered/refused union. Generated-language error envelopes retain the decision error type. Arrays, optional fields and nested evidence work with generative models; native providers require flat required scalar fields. Streaming emits only complete evidence. Label-only examples never manufacture distributions. For reranking, evaluate query/document pairs with the same score rubric, retain candidate IDs outside the model, and sort by relevance.value. Runnable decision-evidence examples are under src/examples/cpp/generation/. ## Typesafe / Jev The typesafe provider supports required boolean, class, and ordered rubric score outputs. Use score "Low, Medium, High" to declare a rubric. Numeric bounds never define a Score rubric; ordinary numbers, freeform strings, optional outputs, arrays, nesting, media, tools, and sampling controls are rejected before transport. Set provider trueThreshold (or true_threshold) to a finite value in [0,1], default 0.5. Boolean conversion uses noul >= threshold; this policy is local and never sent. Choice returns the selected label without a confidence cutoff. Use boolean(true "Core task blocked", false "Routine request") and class label descriptions for criteria. Fluent describe_values / describeValues / DescribeValues keeps the same field value type. C++ uses valueDescriptions on its existing field descriptors. Other providers receive readable prompt and schema descriptions. The separate native client exposes system_one / systemOne / SystemOne and list_models / listModels / ListModels. Native probabilities remain unchanged. Score returns a fractional zero-based rubric position: convert scales explicitly in application code. Entries may be text, structured JSON objects/arrays, or null. Choice allows 1–255 labels; Score requires 2–10 rubric levels. The service context limit covers state, questions, and criteria; Ax never truncates or pretends to count native tokens exactly. Choice and Score probabilities must be finite values in [0,1], match the criteria keys, and sum to one within an inclusive 0.01 tolerance. Totals of 0.99 and 1.01 are accepted with an allowance for floating-point summation error. Ax preserves the returned probabilities without renormalizing them. The default model is jev-latest. Use API keys or renewable credential callbacks, the shared HTTP transport, retry settings, timeout, and cancellation. Native model discovery is separate from configured Ax model aliases. Typed native answers retain question names; only TypeScript can infer literal question keys and Choice-label unions at compile time. Other languages use their native typed maps/records/enums. Typesafe-only balancers propagate the output-schema requirement. Mixed pools retain ordinary prompts and select Typesafe only when the actual request already has a supported schema. Unsupported requests remain excluded during fallback and degradation. Typesafe has no token streaming; the provider returns one completed result through its stream interface. Runnable signature, native criteria/scoring, and two-program hybrid examples are under src/examples/cpp/generation/. See https://axllm.dev/cpp/examples/generation/. ## OpenAI Decisions The openai-decisions provider uses /v1/decisions with gpt-6-luna by default. Required boolean fields become predicates; required class fields become choices; score fields become labelled native rubrics. Use normal AxGen signatures with (evidence) to retain probabilities, provider-selected values, native confidence, and trusted provenance; use (evidence, refusal return) for an answered/refused union. See the Typesafe/Jev skill for decision evidence and reranking across both providers. trueThreshold (or true_threshold) defaults to 0.5, is finite in [0,1], and uses an inclusive comparison. Field value descriptions become criteria instructions or choice descriptions. Raw answers remain in providerMetadata.openaiDecisions.answers; usage uses existing program APIs. Use `axllm::openai_decisions` for explicit provider request shapes and native JSON input. Its create operation (Go: Create) accepts input and an ordered questions array. Each question has string instructions and an optional unique string name; unnamed answers retain name: null. Predicate returns probability. Choice requires 2-255 string or boolean choices and returns choice, confidence, and probabilities. Score requires 2-10 explicitly labelled levels and returns a fractional zero-based score, confidence, and probabilities. Handle type: refusal before accessing answer values. Preserve full model, answer order, and usage. Core validates response names, kinds, rubric membership, finite bounds, and distributions summing to one within an inclusive 0.01 tolerance. Probabilities are never normalized. Native input is text or user messages containing input_text and inline input_image data URLs (up to 128 images). Hosted image URLs, audio, non-user roles, and tool items are rejected. Signature prompt/history becomes role-labelled text evidence in one user message. Numeric bounds never define a score rubric; optional, numeric, nested, array, freeform, tool, and sampling requests are rejected before transport. Use api_key/apiKey or renewable credential providers with shared timeout, retry, and cancellation. Fresh credential headers override static/custom headers case-insensitively on every attempt. Inherited and per-call cancellation both apply. The endpoint returns a completed result without token streaming. Public signature and native examples are under src/examples/cpp/generation/. ## Named Deployment Profiles - The first `ai` / `NewAI` factory argument selects deployment behavior. The model option selects a model only inside that deployment; never infer request rules from a vendor-looking model ID. - `openai` is the official OpenAI deployment. `openai-compatible` is the conservative custom-endpoint profile and requires an explicit base URL. Unknown profile names are errors. - A Together-hosted DeepSeek model uses the `together` profile's URL, authentication, reasoning fields, and effort mapping. Native DeepSeek `thinking` fields apply only to the `deepseek` profile. - Verified DeepSeek, Grok, Groq, Cerebras, and DeepInfra model rules default an omitted thinking level to logical `max`, mapped to the strongest documented deployment effort. - Send `none` only where the selected deployment and model document reasoning disablement. Unsupported levels fail before network I/O; dynamic Hugging Face Router routes remain conservative. - Structured output is an ordered model-aware capability: `native`, `function`, and `json_object`. Exact caller model metadata overrides the first matching profile rule, which overrides the profile default. - An explicit unsupported structured-output mode fails before transport. `structuredOutputs` / `structured_outputs` remains the compatibility alias for native JSON Schema only. - The exact Vertex `google/gemma-4-26b-a4b-it-maas` rule prefers `json_object`, excludes native schema, defaults thinking to `max`, writes nested `enable_thinking`, and extracts/replays `reasoning_content`. Unknown Vertex models stay conservative. - Use named factories for Azure OpenAI, Cohere, DeepSeek, DeepSeek Responses, Mistral, Reka, Grok, routers, hosted inference, and configurable runtimes. Profile-only branded client constructors were removed. - Retained client classes are transport/runtime boundaries: OpenAI-compatible Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini GenerateContent. Build ordinary applications through the named factory. - A profile without a base URL of its own (`openai-compatible`, `databricks`, `amazon-bedrock`, `vertex-ai`, ...) needs one from the caller: `ai(...)` fails with TypeScript's `<Name> requires apiURL` instead of sending the key to another host. Built directly without a base URL, the OpenAI-compatible client class talks to `https://api.openai.com/v1` with the conservative `openai-compatible` profile. For OpenAI itself use the `openai` factory, which applies OpenAI's model catalog. - Environment credentials stay with their own provider: `OPENAI_API_KEY` and `OPENAI_BASE_URL` are read only for `openai`, `openai-responses` and `openai-compatible`; `ANTHROPIC_API_KEY` and `ANTHROPIC_BASE_URL` only for `anthropic`; the Google key variables only for `google-gemini`. Any other provider needs an explicit API key or a credential provider. - Sampling follows TypeScript. A client starts from its provider's defaults: temperature 0, or temperature 0.7 and top-p 1 for `openai-responses`, and none for the other Responses profiles. The client's model config, a model key's config and then the request's merge over them. The catalog marks the sampling parameters a model rejects (temperature, top-p, top-k and the presence/frequency penalties): a default for one is never sent; an explicit value is sent when the model accepts it for that request's reasoning effort (GPT-5.1-5.4 while reasoning is off, their default; GPT-5.5 and 5.6 with effort `none`) and otherwise dropped with a one-time warning naming the setting and the model. A model the catalog marks `temperatureOne` still takes an explicit temperature of 1. The o-series still get the token limit and `n`; a profile without model info uses OpenAI's for an exact o-series name. - Claude Haiku 5.5 (claude-haiku-5-5): adaptive thinking defaults to medium effort. thinkingTokenBudget none sends thinking.type disabled, rejecting xhigh/max effort. Other levels use adaptive thinking and effort. Forced tool choice, native JSON Schema, and function-based structured output are supported. Keep signed and opaque thinking blocks for replay and history append-only, including later system messages on Vertex. Context is 1M tokens with up to 128K output. Input/output cost per million is $0.10/$0.50 through 100K total input (including cache reads and writes), and $0.50/$2.50 above it. - Claude Sonnet 5.5 (claude-sonnet-5-5): thinkingTokenBudget none sends thinking.type between_tools; it hides the updates but retains signed and redacted blocks for replay. Explicit xhigh/max effort with thinking off is rejected; high or below is accepted. The default effort is left to the provider. Native structured output is supported, explicitly forced tool choices are rejected, and later system messages stay in place (including on Vertex). - Anthropic: Opus 4.7 and later, Opus 5, Fable 5 and Sonnet 5 take only temperature 1 (no top-p or top-k); the other Claude models take every value with thinking off, and while thinking only temperature 1, top-p of 0.95 or above, and no top-k, and never temperature with top-p (an explicit top-p goes alone in place of the default temperature; an explicit temperature wins over it with a warning). The default temperature 0 is sent only without thinking. Gemini: the server-managed Flash models ignore temperature, top-p and top-k; the Gemini API rejects the penalties; Gemini 3 raises a temperature below 1 to 1, warning once for an explicit value. Vertex keeps its request shape for both and warns about a value it drops. - Provider descriptors and conformance fixtures are generated from the shared profile manifest. Do not add provider-name switches or cross-profile model normalization in a generated package. ## Vertex And Prompt Caching - Configure Gemini or Anthropic Vertex mode with `projectId` / `project_id` and `region`; optionally select a Vertex endpoint with `endpointId` / `endpoint_id`. - Use `credentialProvider` / `credential_provider` for expiring Vertex and cloud tokens. It receives profile, operation, method, and URL on every attempt; its headers override static authentication. - Credential callbacks cover chat, stream, embeddings, Responses, transcription, speech, and retries. Callback errors stop before transport, and completed 401/403 generation responses are not replayed automatically. - Keep ADC and cloud SDK dependencies host-owned: obtain or refresh the token inside the callback. A required-auth profile accepts either a static key or the callback. - Core resolves `global`, `us`, `eu`, and regional Vertex hosts. An explicit `baseUrl` / `base_url` takes precedence. - `beta` on a call routes that Vertex call onto `v1beta1`, and `beta: false` keeps it on `v1` when the client sets `beta`, as in TypeScript. - On Vertex, `gemini-embedding-2` embeds through `:embedContent` at the `global` location whatever `region` is set. Each call embeds exactly one text, because Vertex fuses a request's texts into one vector, and sends no task type, which Vertex ignores for this model; put task instructions in the text instead. Other embedding models and `endpointId` / `endpoint_id` deployments keep the regional `:predict` call. - OpenAI GPT-5.6 Chat explicit caching is opt-in through `contextCache` / `context_cache` or message/function cache flags. Use `promptCacheKey` / `prompt_cache_key` for stable affinity; `sessionId` / `session_id` is the fallback. - As in TypeScript, every OpenAI Responses request sends `prompt_cache_key`: the `promptCacheKey` / `prompt_cache_key`, else the `sessionId` / `session_id`, the call's before the client's. Chat Completions sends it only with GPT-5.6 caching. - Normalized usage separates uncached prompt, cache-read, and cache-creation tokens. `get_model_cost` / target equivalent uses the shared model catalog, including cache-write pricing and long-context thresholds. - Start with the OpenAI prompt-caching and Vertex Gemini examples under `examples/`. Scripted AxAI fixtures verify routing without live credentials. ## Request Timeouts - With contextCache enabled, cached signature inputs form a stable user-message prefix; dynamic inputs follow separately. Agent stages mark stable inputs cached and keep runtime guidance and action history dynamic. - `timeoutMs` on a chat, stream or embed call bounds the wait for the response headers in milliseconds, as TypeScript's per-call `timeout` does. In the client's options it applies to every call. A request whose response has not started in time fails with `AxAIServiceTimeoutError` (`Request timed out after <N>ms`). The request layer does not retry it, and AxGen retries it as an infrastructure error. Once the response starts, the body reads as it did before. AxGen and agent forwards pass `timeoutMs` to every model call. - A per-call `timeout` is milliseconds. `timeoutMs` remains an alias and wins when both are supplied. Model-call timeouts are not forwarded to the agent runtime; configure execution deadlines on the runtime itself. The client's constructor timeout stays in seconds. ## Transport Errors - A connection that is refused, reset, or closed before a response raises `AxAIServiceNetworkError` with TypeScript's message, `Network Error: <cause>`. The client's own timeout raises `AxAIServiceTimeoutError` (`Request timed out after <N>ms`, the timeout in milliseconds). AxGen retries both as infrastructure errors. - libcurl's message stays the error's `cause()`. ## Request Retries - As TypeScript's apiCall does, chat, embed, context-cache and Typesafe requests, and a stream's request, go out again after a network failure or a status the retry config lists (`retryableStatusCodes`, by default 500, 408, 429, 502, 503, 504 and 529), up to `maxRetries` times (default 3). - The wait is `initialDelayMs * backoffFactor ** attempt` (1 s doubling by default), at most `maxDelayMs` (60 s), times a jitter of 0.75 to 1.25. A status response's `Retry-After`, in seconds or as an HTTP date, replaces it when it is no longer than `maxDelayMs`. - The call's `retry` options replace the client's. `retry: { maxRetries: 0 }` sends each request once. - A 401 or 403, a timeout the request ran out of, and an aborted request are never retried here. A stream whose response began is not retried either: a failure to read its first event surfaces. A first event that carries a listed status (Anthropic's `overloaded_error`) goes out again, with its own budget and without jitter. ## Routing And Balancing - Use the multi-service router when a logical model key selects a configured service or concrete model. It combines model lists; it does not learn from outcomes. - Use `ProviderRouter` for capability-based selection and optional media degradation. When the selected provider supports images, preserve every native image part with its payload, MIME type, detail level, cache and optimization hints, alt text, and ordering with surrounding text. - Native files retain filename, MIME type, data, cache flags, extraction metadata, and order through provider/model selection and later conversation turns. Existing extracted text is used only when the selected provider cannot consume the file. - For unsupported files, configure a file-to-text callback or choose degradation, skip, or error policy. An empty extraction result is valid; extraction failures stop before the provider request. Python, Go, and Java accept fileToText in router processing options; C++ exposes file_to_text, and Rust exposes with_file_to_text and with_processing. - Inline PDF inputs should include filename, mimeType, and base64 data. See the public native-file-routing generation example for this language; it uses the ordinary generator and a real provider. - Use the default `AxBalancer` for deterministic ordered/metric failover with its existing retry policy. - Opt into `AxBalancerAdaptiveStrategy` only for operational routing among application-approved equivalent aliases. It learns transient reliability and successful latency, combines them with estimated cost and a deadline, and explores with Thompson sampling. - Put centralized decision state in an `AxBalancerStatsStore`. The routing-event callback is best-effort analytics and observability, not a state replication mechanism. - Shared stores require non-empty, unique, stable route keys. Use slices to isolate workflows, tenants, or traffic classes without putting prompts, responses, raw errors, or sensitive identifiers in keys or events. - Adaptive balancing does not measure answer quality or semantically choose a model. Only group routes that the application already accepts as substitutes. - Generated provider streams are incremental and closeable. Retry or failover is allowed only before the first content event; later failures surface without replay, and adaptive latency is recorded at the first chunk. - Start with `examples/adaptive_balancer_no_key` for store/reducer syntax, then use the cataloged provider-backed adaptive-balancer example for a complete two-route setup. ## Astra Session Work Select `gpt-6-astra` through the ordinary OpenAI factory. The adapter chooses Responses automatically; existing model defaults are unchanged. Use low reasoning and standard processing. Portable minimal reasoning maps to low; none is rejected. EU residency does not support priority processing. Keep applications on their generation, agent, and flow entrypoints. Declare only independent tools as background; ordinary and imported MCP tools stay blocking unless the application explicitly changes their declaration. A promise, thread, or MCP hint is not a background declaration. Set `asyncMode` to `off` for the ordinary tool loop; chat-only services retain that loop automatically. Declared-background native agent tools retain the imported MCP schema, handler, namespace, and raw result. Discovery must expose the tool before the model can call it. Invalid arguments are corrected before handler execution; the responder waits for the incorporated result. Native calls appear in action logs and must not be repeated through actor code. Owned child agents inherit selected MCP clients at delegation. Parent stages keep their own clients; none or an empty namespace list passes no parent clients. Explicit child context wins over inherited context. Each run refreshes protocol modules without serializing live client handles into model requests. Cancellation propagates through a delegated child into its pending MCP tool; completed child work is not replayed. Imported MCP tools forward cancellation to context-aware transports, including built-in HTTP. Custom transports using the older send method receive cancellation checks before and after their call; noncooperative work may finish later and its result is discarded. Cancellation does not undo an external action or replay a request. MCP host policy applies to native background calls too. Configure authorizeToolCall in Python, Go, and Java client options, or set_tool_authorizer on C++ and Rust clients before exposing their tools. The callback receives the client and call metadata; returning false denies the call before a tool request is sent. Use shared application policy state when permissions must change during a run. Register child agents before running the parent: add_child_agent(namespace, name, child) in Python/C++, AddChildAgent in Go, addChildAgent in Java, and with_child_agent in Rust. Registered children are available automatically as namespaced actor calls, such as team.researcher({question}). Calls use discovery, validation, and invocation accounting. Child invocation remains serialized on the owning run thread and owns a separate conversation. Retained callbacks reject calls after the run closes. Controls target paths such as root/team.researcher/executor. Child results return through the parent invocation log, and parent usage includes a children section. Attach the language-native run controller through forward options for steering, reasoning changes, cancellation, and lifecycle events. Queued and applied are different states. HTTP applies updates at a response boundary; an optional host WebSocket enables native steering. Do not manage response IDs, socket messages, or tool-result submission in application code. As in TypeScript, an update queued while a request is in flight applies when the next step starts. If that request gave the final answer, the run takes one more step to apply it, and the answer comes from that step; a steer stays in the conversation for the steps after it. Each model request opens its own native session with the whole conversation and closes it when that request's response completes, as in TypeScript. A correction or a later step opens a fresh session, which applies the run's updates again. A stream sends a session's partial output as it arrives: each response streams like a plain stream, a later response starts a new version, and the completed response adds only what was not sent. A forward does not apply streaming assertions. A provisional answer is not successful completion while started tools remain unresolved. Cancellation closes the session, reports unresolved call IDs, and retains unresolved started calls in tool traces and native agent action logs; it cannot undo an external action. Handlers may cooperate through the invocation cancellation context. Late results from noncooperative work must not change a closed run or trigger replay. Java, C++, and Rust WebSocket adapters track activity when frames arrive. Consuming buffered events does not reactivate a completed response. When no response is active, steering is queued for the next response; an active successor can still receive native steering. Observe lifecycle timing instead of assuming native application. All five session adapters validate completed raw arguments against the shared Core validator before invoking handlers, including local references, unions, nested schemas, additional properties, and numeric/string/array constraints. Raw schema patterns use shared flagless ECMAScript semantics, including UTF-16, lookarounds, named captures, and backreferences. Invalid arguments enter correction; step exhaustion fails the run. Independent flow nodes use owned program and client workers. Built-in providers, routers, and balancers supply factories; custom implementations without them run the entire group serially and emit a flow_parallel_fallback trace. Rust does not require Send/Sync on the existing client trait. Rust nested flows and custom AxExecutableProgram implementations use execute_program; an optional AxOwnedProgramFactory constructs state on its worker. Workers deliver events and results to the owner, which merges each successful step's changes in plan order, so a group ends as running its steps one after another would. On group failure, cancellation preserves completed diagnostics and discards late deliveries. Use the provider-backed Astra examples under `src/examples/cpp/generation/`, `short-agents/`, and `flows/`. All-five generated parity remains under verification in the shared-session AxIR backlog; do not infer full agent, parallel-flow, or transport parity from these examples alone. ## Relevant API Surface - AxAI: `axllm::ai`, `axllm::openai_decisions`, `axllm::AxAIOpenAIDecisionsClient`, `axllm::typesafe`, `axllm::AxAITypesafeClient`, `axllm::AxCancellationToken`, `axllm::AxAIServiceAbortedError`, `axllm::get_supported_ai_models`, `axllm::AxCredentialRequest`, `axllm::AxCredentialProvider`, `axllm::AIClient::owned_worker_factory`, `axllm::AxChatSession`, `axllm::AxStreamHandler`, `axllm::OpenAICompatibleClient`, `axllm::OpenAIResponsesClient`, `axllm::GoogleGeminiClient`, `axllm::AnthropicClient`, `axllm::AxUsageContext`, `axllm::AxUsageEvent`, `axllm::AxUsageObserver`, `axllm::set_usage_observer`, `axllm::AxRuntimeHooks`, `axllm::AxRateLimitInfo`, `axllm::AxRateLimiter`, `axllm::AxTracer`, `axllm::AxMeter`, `axllm::AxGlobals`, `axllm::set_rate_limiter`, `axllm::set_tracer`, `axllm::set_meter`, `axllm::AxBalancer`, `axllm::AxBalancerAdaptiveStrategy`, `axllm::AxBalancerStatsStore`, `axllm::AxInMemoryBalancerStatsStore`, `axllm::create_balancer_route_stats`, `axllm::update_balancer_route_stats`, `axllm::sample_balancer_route_health`, `axllm::MultiServiceRouter`, `axllm::ProviderRouter` ## Guardrails - Start from package examples for exact native syntax before inventing a new call shape. - Use `provider-api` examples only when the user explicitly has provider credentials available. - Use `no-key` examples for deterministic local checks and provider request mapping. - Treat AxIR as the source of generated package truth: if package docs disagree with source code, update the compiler and regenerate packages. - Do not copy repo-maintainer skills from `tools/*/skills/` into user packages.