ax-host

name	ax-host
description	Use when modifying the trusted host process — server orchestration, message routing, IPC handler, request lifecycle, event streaming, file handling, plugin loading, or agent delegation in src/host/

Overview

The host subsystem is the trusted half of AX. It runs the HTTP server (OpenAI-compatible API over Unix socket and optional TCP), routes inbound/outbound messages through security scanning and taint tracking, dispatches IPC actions from sandboxed agents to provider implementations, manages the agent process lifecycle (spawn, stdin, stdout, cleanup), streams real-time completion events via SSE, handles file uploads/downloads, loads third-party plugin providers, and orchestrates sub-agent delegation with depth/concurrency limits.

Key Files

File	Responsibility
src/host/server-local.ts	HTTP server composition root (local mode): Unix socket + TCP lifecycle, channel connect/disconnect, legacy migration, file upload/download, OAuth callbacks, graceful drain. Delegates shared init to server-init.ts and shared handlers to server-request-handlers.ts
src/host/server-init.ts	Shared initialization for both server-local.ts and server-k8s.ts: initHostCore() sets up storage, routing, taint budget, agent dirs, template seeding, skills seeding, admins, IPC, CompletionDeps, delegation, orchestrator, agent registry. Also loads plugin MCP servers on startup via reloadPluginMcpServers(), loads database MCP servers via loadDatabaseMcpServers(), and creates McpConnectionManager
src/host/server-request-handlers.ts	Shared HTTP handlers: handleModels, handleCompletions (body parsing + streaming + non-streaming via runCompletion callback), handleEventsSSE (EventBus-based SSE), createSchedulerCallback factory
src/host/server-admin-helpers.ts	Pure admin functions: isAgentBootstrapMode, isAdmin, addAdmin, claimBootstrapAdmin (used by server-local.ts, server-completions.ts, IPC handlers)
src/host/server-webhook-admin.ts	Shared webhook + admin handler factories: setupWebhookHandler, setupAdminHandler
src/host/server-completions.ts	Completion processing, workspace setup, history loading, image extraction, agent spawning, response parsing. Container sandboxes (Docker/Apple) use three-phase orchestration: provision (network) → run (no network) → cleanup (network)
src/host/server-channels.ts	Channel ingestion, message deduplication, thread gating/backfill, emoji reactions, attachment handling, per-message agent routing (resolveAgentForMessage), thread ownership tracking (ThreadOwnershipMap), response prefix for personal agents in shared channels (maybeAddResponsePrefix)
src/host/server-files.ts	File upload/download API, workspace file storage, MIME type handling
src/host/server-http.ts	HTTP utilities, SSE chunking, body reading, error responses
src/host/server-lifecycle.ts	Stub — lifecycle duties now distributed to server-local.ts and server-k8s.ts
src/host/event-bus.ts	Typed pub/sub for real-time completion observability, global + per-request listeners
src/host/chat-termination.ts	logChatTermination(reqLogger, { phase, reason, sandboxId?, exitCode?, details? }) — unified chat_terminated error-level event called from every host-side site where a chat turn dies abnormally. Plus logChatComplete(reqLogger, { sessionId, agentId?, durationMs, phases?, sandboxId?, tokens? }) — paired success-side chat_complete info-level event so every chat (happy or sad) produces exactly one canonical greppable line. Plus createWaitFailureTracker() for retry loops — emits chat_terminated EXACTLY ONCE per terminated chat (not per attempt). See ax-logging-errors skill for conventions
src/host/router.ts	Inbound scan + taint-wrap + canary inject; outbound scan + canary check
src/host/ipc-server.ts	Unix socket server, IPC action dispatch, Zod validation, taint budget gate, heartbeat, event emission. Concurrent IPC call fix (misrouted responses on shared socket). proxy.sock race prevention (await IPC server listen). agent_response timeout handling without crashing
src/host/ipc-handlers/browser.ts	Browser automation IPC handlers
src/host/ipc-handlers/delegation.ts	Agent delegation with depth/concurrency limits, zombie counter prevention
src/host/ipc-handlers/governance.ts	Proposal list/review IPC handlers
src/host/ipc-handlers/identity.ts	Identity read/write/propose IPC handlers (backed by DocumentStore)
src/host/ipc-handlers/image.ts	Image generation handler, in-memory image storage per session
src/host/ipc-handlers/llm.ts	LLM call IPC handler with event emission
src/host/ipc-handlers/memory.ts	Memory CRUD IPC handlers
src/host/ipc-handlers/orchestration.ts	Agent orchestration IPC handlers (status, list, tree, message, poll, interrupt, timeline)
src/host/ipc-handlers/plugin.ts	Plugin status/list queries (host-internal actions)
src/host/ipc-handlers/scheduler.ts	Scheduler job management IPC handlers
src/host/ipc-handlers/sandbox-tools.ts	Sandbox tool IPC handlers (sandbox_bash, sandbox_read_file, sandbox_write_file, sandbox_edit_file) and audit gate protocol (sandbox_approve, sandbox_result) for in-container tool execution with host approval
src/host/ipc-handlers/skills.ts	Skill read/list/propose/import/search IPC handlers + credential_request handler (backed by DocumentStore)
src/host/skills/get-agent-skills.ts	Live per-agent skill state derivation. getAgentSkills() / getAgentSetupQueue() walk the workspace repo + diff frontmatter against stored creds / approved domains. Backed by the (agentId, HEAD-sha) snapshot cache
src/host/skills/snapshot-cache.ts	Bounded LRU snapshot cache keyed on ${agentId}@${headSha}. Supports invalidateAgent(agentId) for post-push cache-bust
src/host/skills/hook-endpoint.ts	HMAC-verified post-receive hook. On a signed push, invalidates the agent's cached snapshots so the next live read is fresh. No other side effects
src/host/ipc-handlers/web.ts	Web fetch/search IPC handlers
src/host/ipc-handlers/workspace.ts	Workspace read/write/list IPC handlers
src/host/ipc-handlers/describe-tools.ts	describe_tools meta-tool handler — schema lookup by catalog name; names: [] returns the full directory
src/host/ipc-handlers/call-tool.ts	call_tool meta-tool handler — dispatches MCP or OpenAPI by catalog entry, handles _select jq projection + auto-spill, returns structured {error, kind} on failure
src/host/ipc-handlers/openapi-dispatcher.ts	Default HTTP dispatcher used by call_tool for dispatch.kind === 'openapi'. Path/query/header/body routing, 4 auth schemes (bearer/basic/api_key_header/api_key_query), URL rewrites, credential redaction in failure logs
src/host/tool-catalog/	Session-scoped tool catalog: registry, MCP adapter, OpenAPI adapter (pure, takes dereferenced spec), jq projection, per-(agentId,HEAD-sha) cache
src/host/skills/catalog-population.ts	Walks the skill snapshot, builds the catalog from mcpServers[] + openapi[], pushes Diagnostic entries on failures and wide-surface advisories
src/host/skills/openapi-spec-fetcher.ts	Default fetchOpenApiSpec factory — HTTPS URLs go direct to @apidevtools/swagger-parser, workspace-relative paths resolve through the bare git repo with traversal guards
src/host/diagnostics.ts	Per-turn ring-buffered DiagnosticCollector — populate failures and wide-surface advisories emitted as event: diagnostic SSE frames at end-of-turn
src/plugins/mcp-manager.ts	McpConnectionManager — unified MCP tool discovery and routing across all MCP sources
src/host/inprocess.ts	In-process fast path — runs LLM orchestration loop directly in host process (no pods, no IPC, no proxy). Uses FastPathDeps (including McpConnectionManager), FastPathRequest, FastPathResult. AsyncLocalStorage for per-turn context isolation
src/host/tool-router.ts	Tool router for in-process fast path — routes tool calls to MCP providers (unified via McpConnectionManager), lazy file I/O, or sandbox escalation. Unified MCP methods: resolveServer(), mcpCallTool(), resolveHeaders(). Per-turn limits (FAST_PATH_LIMITS). Exports routeToolCall(), ToolRouterContext, ToolResult
src/host/sandbox-manager.ts	Sandbox session manager — tracks session-bound sandbox pods for cross-turn escalation from fast path. CRUD on DocumentStore with TTL (30min default, 1hr max)
src/host/agent-registry.ts	Enterprise agent registry (registry.json), lifecycle management. AgentRegistryEntry has displayName, agentKind ('personal'/'shared'), admins[]. findByKind() filters by agent kind
src/host/agent-registry-db.ts	Database-backed agent registry for PostgreSQL (Kysely, runs own migration). Migration 003 adds display_name and agent_kind columns
src/host/server-admin.ts	Admin API endpoints (agent management, config, diagnostics). Includes admin API endpoints for MCP server management under /admin/api/agents/:id/mcp-servers and the phase-5 skill setup endpoints under /admin/api/skills/setup and /admin/api/credentials/requests
src/host/server-admin-skills-helpers.ts	Helper for POST /admin/api/skills/setup/approve. Validates the request body against the live setup card (via getAgentSetupQueue), then writes credentials (legacy credentials.set + tuple-keyed skill_credentials), approves pending domains by writing skill_domain_approvals rows, invalidates the agent's snapshot cache, and reads the fresh SkillState via getAgentSkills. Narrow ApproveDeps interface avoids circular imports with server-admin.ts
src/host/credential-request-queue.ts	Phase-5 in-memory queue of ad-hoc credential requests. Seeded by credential.required event-bus subscriptions; drained when POST /admin/api/credentials/provide succeeds. Exposed to the dashboard via GET /admin/api/credentials/requests
src/host/server-k8s.ts	Unified host pod process for k8s deployment. Delegates shared init to server-init.ts. Keeps k8s-specific: NATS connection (work dispatch only), /internal/* routes (ipc, llm-proxy, workspace), web proxy with MITM CA, stagingStore, activeTokens registry. Uses server-request-handlers.ts for completions/models/scheduler. Use server-local.ts for local dev
src/host/server-chat-api.ts	Chat API handler — serves /v1/chat/sessions endpoints for chat UI thread list and history
src/host/server-chat-ui.ts	Chat UI static file serving — serves built chat UI from dist/chat-ui/ at root path
src/host/session-title.ts	Auto-generate session titles from first user message using fast LLM model
src/host/llm-proxy-core.ts	Shared LLM credential injection and forwarding — used by both Unix socket proxy (proxy.ts) and HTTP route (/internal/llm-proxy in server-k8s.ts)
src/host/oauth-skills.ts	OAuth PKCE flow for skill credentials — manages pending flows (start → callback → token exchange → store), handles token refresh
src/host/admin-oauth-providers.ts	AES-256-GCM encrypted admin-registered OAuth provider CRUD. Key derived from AX_OAUTH_SECRET_KEY (preferred, 32 hex bytes) or config.admin.token (fallback, requires ≥16 chars). clientSecret never returned on read.
src/host/admin-oauth-flow.ts	PKCE pending-flow map (15-min TTL, single-use) + resolveCallback that exchanges code via AbortSignal.timeout(15s), writes access_token at declared credential scope, stores refresh blob at <envName>__oauth_blob, invalidates the agent's snapshot cache so the next live read picks up the new credential. Returns `{matched:false}
src/migrations/admin-oauth-providers.ts	Kysely migration for admin_oauth_providers table (name PK, client_id, encrypted client_secret blob, metadata).
src/host/web-proxy-approvals.ts	Web proxy approval coordination via event bus — replaces old in-memory promise map pattern. Works across stateless replicas (in-process for local, NATS for k8s)
src/host/workspace-release-screener.ts	Release-time screening for skill files and binaries — inspects workspace changes before GCS commit
src/host/delivery.ts	Delivery resolution for cron/heartbeat responses (CronDelivery handling)
src/host/event-console.ts	Real-time event display with color-coded output
src/host/history-summarizer.ts	Recursive conversation summarization with LLM
src/host/memory-recall.ts	Memory recall integration with embeddings for context injection
src/host/oauth.ts	OAuth token management (pre-flight + reactive 401 retry)
src/host/server-webhooks.ts	Inbound webhook handler with LLM-powered transforms
src/host/webhook-transform.ts	LLM-powered webhook payload transformation
src/host/plugin-host.ts	Plugin lifecycle manager, integrity verification, process management, IPC proxy
src/host/plugin-lock.ts	Plugin manifest pinning, SHA-512 integrity hashing, lock file I/O
src/host/plugin-manifest.ts	Plugin capability schema, validation, human-readable formatting
src/host/proxy.ts	Credential-injecting Anthropic forward proxy, OAuth 401 retry
src/host/web-proxy.ts	HTTP forward proxy for agent outbound HTTP/HTTPS. HTTP forwarding + HTTPS CONNECT tunneling, private IP blocking (SSRF), canary token scanning on request bodies, audit logging. Listens on Unix socket (container sandboxes) or TCP port (subprocess/k8s). Opt-in via config.web_proxy
src/host/taint-budget.ts	Per-session taint ratio tracking, action gating (SC-SEC-003)
src/host/provider-map.ts	Static allowlist mapping config names to provider modules (SC-SEC-002), plugin registration runtime allowlist. mcp category has none and database entries
src/host/registry.ts	Loads and assembles ProviderRegistry from config; three loading patterns (simple, manual-import with deps, custom); plugin host integration. MCP provider requires { database, credentials } deps

Request Lifecycle (server-local.ts + server-completions.ts)

HTTP Completion Request

1. HTTP Handler -- Parse OpenAIChatRequest JSON, validate session_id, derive stable session ID (explicit -> user field -> random UUID), extract user/conversationId
2. Router + Inbound Scan -- Build InboundMessage, call router.processInbound() (scan, taint-wrap, canary inject, enqueue), emit completion.start and scan.inbound events
3. Message Dequeue -- Dequeue by ID (not FIFO) from MessageQueue
4. Workspace Setup -- Create workspace dir, load identity from committed git state (or seed new repo via seedAxDirectory), build conversation history (DB-persisted for persistent sessions, client-provided for ephemeral), prepend parent channel context for thread sessions
5. Credential Proxy (claude-code only) -- Refresh OAuth pre-flight, start Anthropic credential-injecting proxy, pass proxy socket to agent
6. Web Proxy (opt-in, config.web_proxy) -- Start HTTP forward proxy per completion. Container sandboxes get a Unix socket (web-proxy.sock in IPC dir); subprocess gets a TCP port. Proxy enforces private IP blocking, canary scanning, and audit logging.
7. Agent Spawn -- Build spawn command with runner args, spawn with stdio pipes, write JSON payload to stdin (history, message, taintRatio, profile, requestId, identity), collect stdout/stderr concurrently
8. Outbound Scanning + Response Processing -- Call router.processOutbound(), parse agent response (structured JSON with __ax_response or plain text), extract image_data blocks -> save to workspace/files/ -> convert to file refs, drain generated images
9. Persistence + Cleanup -- Persist conversation turns, attach file refs, clean up workspace/proxy, emit completion.done event

Channel Message Ingestion (server-channels.ts)

1. Deduplication: TTL-based processedMessages map keyed by channelName:messageId
2. Thread Gating: Threads only processed if SOUL.md exists
3. Thread Backfill: Fetch parent message + thread root, build context
4. Bootstrap Gate: Require admin claim and IDENTITY.md before processing
5. Attachment Handling: Download image attachments, embed as image_data content blocks (no disk round-trip)
6. processCompletion Call: Pass preprocessed message, userId, replyOptional flag
7. Outbound: Upload generated images + file refs to channel

Event Bus (event-bus.ts)

Typed pub/sub for real-time completion observability.

• Synchronous emit (fire-and-forget) -- never blocks the hot path
• Global listeners: receive all events (max 100)
• Per-request listeners: scoped to requestId (max 50 per request)
• Event types: dot-namespaced (e.g. completion.start, llm.done, scan.inbound, status)
• No secrets: event.data never contains credentials or sensitive info
• SSE endpoint: GET /v1/events?request_id=...&types=... with 15s keepalive

Status SSE Events

server-completions.ts emits structured status events via the EventBus during completion processing. server-request-handlers.ts forwards these as SSE named events (event: status\ndata: {...}\n\n) to chat UI clients.

Status event structure:

{ "type": "status", "requestId": "...", "timestamp": 12345, "data": { "operation": "workspace|pod", "phase": "...", "message": "..." } }

Current status events:

Operation	Phase	Message	When
workspace	downloading	Restoring workspace…	Before workspace mount
workspace	mounted	Workspace ready	After successful mount
pod	creating	Starting sandbox…	Before first agent spawn
pod	retrying	Retrying sandbox (attempt N)…	On agent retry

SSE forwarding (server-request-handlers.ts): Validates that operation, phase, and message are all strings before forwarding. Invalid payloads are logged as status_event_invalid_payload.

MCP Fast Path (In-Process Agent)

The in-process fast path (src/host/inprocess.ts) runs the LLM orchestration loop directly in the host process — no sandbox pods, no IPC, no proxy, no GCS sync. Used for lightweight MCP tool-calling tasks via the MCP provider interface.

Key components:

• inprocess.ts — LLM turn loop: builds tool definitions from MCP provider + built-in file tools, calls LLM, routes tool calls, persists history
• tool-router.ts — Routes tool calls to: MCP provider (callTool), built-in file_read/file_write (using safePath()), or request_sandbox (escalation to full sandbox)
• sandbox-manager.ts — Tracks session-bound sandbox pods for cross-turn escalation. CRUD on DocumentStore with TTL (30min default, 1hr max)

Per-turn limits (FAST_PATH_LIMITS):

Limit	Value
maxToolCallsPerTurn	50
maxTurnDurationMs	5 minutes
maxToolResultSizeBytes	1 MB
maxTotalContextBytes	10 MB

MCP Provider (src/providers/mcp/types.ts): Interface for external tool gateways. Methods: listTools(), callTool(), credentialStatus(), storeCredential(), listApps(). Implementations: none (no-op), database (per-agent HTTP/SSE MCP servers stored in DB with per-server circuit breakers, credential placeholder resolution, and server__tool name prefixing). CRUD helpers: addMcpServer(), removeMcpServer(), listMcpServers(), updateMcpServer(), testMcpServer(). Unified routing via McpConnectionManager (src/plugins/mcp-manager.ts) — tools discovered from plugins, database MCP servers, and default MCP provider in one pass.

File Storage (server-files.ts)

• Endpoints: POST /v1/files?agent=<name>&user=<id> (upload), GET /v1/files/<fileId>?agent=<name>&user=<id> (download)
• Storage: ~/.ax/agents/<name>/workspaces/<userId>/files/<fileId>
• MIME types: image/png, image/jpeg, image/gif, image/webp (10 MB max)
• FileStore: metadata registry (fileId -> agentName, userId) for lookups without query params

IPC Server (ipc-server.ts)

Protocol: 4-byte big-endian length prefix + JSON over Unix socket.

Dispatch pipeline:

1. Parse JSON
2. Validate envelope schema (IPCEnvelopeSchema)
3. Validate action-specific Zod schema (.strict() mode)
4. Step 3.5: Taint budget check -- hard-blocks tainted sessions (except identity_write, user_write, identity_propose)
5. Dispatch to handler function, audit log result

Heartbeat frames: Long-running handlers emit {_heartbeat: true, ts} every 15s to prevent client timeout.

Event emission: Handlers emit events via eventBus (e.g., llm.start, llm.done) for streaming observability.

Plugin System

Plugin Manifest (plugin-manifest.ts)

• Schema validation: name, kind (llm/image/memory/etc.), capabilities
• Capabilities: network (host:port endpoints), filesystem (none/read/write), credentials (injected keys)
• No wildcards, explicit declarations only

Plugin Lock File (plugin-lock.ts)

• ~/.ax/plugins.lock -- JSON file pinning exact versions and SHA-512 hashes
• On every startup, hashes verified before plugin loads

Plugin Host (plugin-host.ts)

• Lifecycle: startAll() -> spawn workers -> verify integrity -> register providers
• Worker Process: fork() with restricted environment (no credentials, minimal env vars)
• IPC Protocol: plugin_call -> plugin_response
• Credential Injection: server-side resolver (plugin never sees credential store)
• Timeouts: startup (10s), call (30s)
• Graceful Shutdown: send plugin_shutdown, wait 5s, force-kill

MCP Connection Manager

MCP servers are registered through two paths — the git-native skills pipeline and database-backed entries:

• McpConnectionManager (src/plugins/mcp-manager.ts): Unified MCP tool discovery and routing — tools from skill-registered servers, database MCP servers, and default provider are discoverable in one pass
• MCP startup (src/plugins/startup.ts): Bootstrap MCP servers from the database + config on host init
• Skill-driven registration: MCP servers declared in a skill's frontmatter are rebuilt per-turn from the live getAgentSkills view. No persistent applier state — the turn-time read is authoritative.
• Admin API: /admin/api/agents/:id/mcp-servers for ad-hoc database-backed entries (not tied to a specific skill)

The old Cowork plugin IPC surface and ax plugin/ax mcp CLI have been retired.

Provider Map (provider-map.ts)

• Built-in Allowlist (_PROVIDER_MAP): static mapping of (kind, name) -> relative/package paths
• Plugin Registration: runtime allowlist (_pluginProviderMap) for Phase 3 plugins
• URL Scheme Guard: post-resolution check ensures all paths are file:// URLs (SC-SEC-002)
• Functions: resolveProviderPath(), registerPluginProvider(), unregisterPluginProvider(), listPluginProviders()
• sandbox category: docker, apple, k8s
• workspace category: git-http, git-local
• skills category: database only

Git-Native Skills Pipeline (src/host/skills/)

Skills live as .ax/skills/**/SKILL.md files in the agent's git workspace. Git is the one source of truth. The host derives each skill's state live from a per-(agentId, HEAD-sha) snapshot of the repo, diffed against the tuple-keyed skill_credentials + skill_domain_approvals tables:

• src/host/skills/get-agent-skills.ts — getAgentSkills(agentId, deps) returns SkillState[]; getAgentSetupQueue(agentId, deps) returns the pending-card payload. Both read through the snapshot cache and compute via the pure helpers in state-derivation.ts.
• src/host/skills/snapshot-cache.ts — bounded LRU keyed on ${agentId}@${headSha}. invalidateAgent(agentId) drops every entry whose key starts with that prefix; used by the post-receive hook and the approve/OAuth callback paths.
• src/host/skills/hook-endpoint.ts — the post-receive hook. HMAC-verifies the payload and calls snapshotCache.invalidateAgent(agentId). No state-store writes, no appliers.

There is no reconcile orchestrator, no skill_states/skill_setup_queue writes, and no startup rehydrate — the retired tables have been dropped (migration skills_003_drop_retired_tables). See docs/plans/2026-04-18-skills-single-source-of-truth-design.md for the full redesign.

Dashboard Admin Endpoints

The host exposes four admin endpoints under /admin/api/ for the dashboard to drive skill setup:

• GET /admin/api/skills/setup — returns pending setup cards grouped by agent. Derived live per agent via getAgentSetupQueue; decorated with hasExistingValue hints from skill_credentials.
• POST /admin/api/skills/setup/approve — atomic approve. src/host/server-admin-skills-helpers.ts owns the validate-all-then-apply-all logic: validates every credential + domain in the body against the live pending card, then (only if all checks pass) writes tuple-keyed rows to skill_credentials (user_id = caller's userId for user-scope, '' sentinel for agent-scope), approves domains by writing skill_domain_approvals rows, invalidates the agent's snapshot cache, and returns the fresh SkillState. Rejects authType: oauth credentials — the OAuth start/callback flow handles those.
• DELETE /admin/api/skills/setup/:agentId/:skillName — dashboard-only dismissal. Reads the live queue via getAgentSetupQueue; when the skill is pending, emits a skill_dismissed audit entry and returns {ok:true, removed:true}. Idempotent (no pending card → removed:false, no audit). The card regenerates on the next load unless the underlying facts change.
• GET /admin/api/credentials/requests — snapshot of the in-memory credential request queue at src/host/credential-request-queue.ts. Seeded by credential.required event-bus subscriptions; drained when POST /admin/api/credentials/provide succeeds.

Phase 6: OAuth Flow

Skill credentials with authType: oauth can be filled via a PKCE-based OAuth flow initiated from the dashboard. Admin-registered providers (stored in admin_oauth_providers) override the frontmatter's public client_id with pre-configured client_id/client_secret — the escape hatch for providers that don't support PKCE-only public flows (e.g. Google confidential web flow).

Endpoints under /admin/api/:

• GET /oauth/providers — list admin-registered providers (never includes client_secret)
• POST /oauth/providers — upsert with Zod-strict body. Audits hasSecret: boolean.
• DELETE /oauth/providers/:name — idempotent.
• POST /skills/oauth/start — validates {agentId, skillName, envName} against the current setup queue, picks up admin-registered overrides, returns {authUrl, state}.

Callback: GET /v1/oauth/callback/:provider (unchanged URL; public endpoint) tries the admin-initiated flow first (src/host/admin-oauth-flow.ts), falls through to agent-initiated (src/host/oauth-skills.ts). Admin callback exchanges the code via the provider's token endpoint (15s timeout), writes the access_token at the card's declared credential scope, stores a refresh blob at <envName>__oauth_blob, and invalidates the agent's snapshot cache so the next turn sees the newly-satisfied credential.

Security invariants:

• clientSecret encrypted at rest with AES-256-GCM; key from AX_OAUTH_SECRET_KEY (32 hex bytes) preferred, derived from config.admin.token as a fallback (requires ≥16 chars).
• No tokens or client_secrets in audit args, logs, or response bodies.
• OAuth state is single-use (consumed even on failure, to prevent replay into the fall-through agent path).
• Provider URL-path must match the pending flow's provider — mismatch is treated as matched-but-failed.

See docs/plans/2026-04-17-phase6-skills-oauth-implementation.md for the detailed plan.

Image Generation (ipc-handlers/image.ts)

• Handler: image_generate IPC action
• Storage: Generated images held in-memory per session in pendingImages map
• Lifetime: After agent finishes, drainGeneratedImages(sessionId) retrieves all images for channel upload + history persistence

Agent Delegation (ipc-handlers/delegation.ts)

• Handler: agent_delegate IPC action
• Depth Tracking: stored in agentId (e.g., delegate-foo:depth=1)
• Limits: maxDepth (default 2), maxConcurrent (default 3)
• Zombie Counter Prevention: activeDelegations++ before any await, decremented in finally block
• Config Override: optional runner/model/maxTokens/timeoutSec passed to processCompletion

Orchestration Subsystem (src/host/orchestration/)

Multi-agent orchestration for parent-child delegation with lifecycle management:

• orchestrator.ts -- Main orchestration engine, coordinates agent delegation and collection
• agent-loop.ts -- Core agent execution loop
• agent-supervisor.ts -- Agent lifecycle supervision and recovery
• agent-directory.ts -- Active agent registration and lookup
• heartbeat-monitor.ts -- Liveness detection for child agents
• event-store.ts -- Audit trail for orchestration events
• types.ts -- Orchestration types (AgentHandle, etc.)

Key IPC actions: agent_orch_status, agent_orch_list, agent_orch_tree, agent_orch_message, agent_orch_poll, agent_orch_interrupt, agent_orch_timeline.

History Summarization & Memory Recall

• history-summarizer.ts -- Recursive conversation summarization via LLM. Enables infinite-length conversations by condensing older turns.
• memory-recall.ts -- Semantic embedding-based memory retrieval. Injects relevant memories into conversation context via src/utils/embedding-client.ts.

Webhooks (server-webhooks.ts + webhook-transform.ts)

• Inbound webhooks: POST /v1/webhooks endpoint for external event ingestion
• LLM-powered transforms: Markdown-based transform definitions convert external payloads to agent prompts
• Configurable: Optional agentId, sessionKey, model, timeoutSec fields per transform
• Config: config.webhooks with enabled, token, path, max_body_bytes, model, allowed_agent_ids

Router (router.ts)

• processInbound(msg): Canonicalizes session ID, generates canary token, wraps content in <external_content> taint tags, records in taint budget, runs scanner.scanInput(), enqueues with canary appended. Returns RouterResult with queued, messageId, canaryToken.
• processOutbound(response, sessionId, canaryToken): Checks canary leakage, scans output, strips canary. Redacts entire response if canary leaked.

K8s HTTP Subsystem (K8s Deployment)

For Kubernetes deployments, k8s sandbox pods communicate with the host over HTTP (not NATS). NATS is used for event bus only in k8s; work dispatch uses HTTP via SessionPodManager. The key HTTP routes on the host (server-k8s.ts):

• POST /internal/ipc — HTTP IPC route. Receives IPC requests from HttpIPCClient in k8s sandbox pods, routes through the same handleIPC pipeline as the Unix socket path. Authenticated via Authorization: Bearer <ipcToken>.
• POST /internal/llm-proxy/* — LLM proxy route. Forwards LLM requests to Anthropic API with credential injection via llm-proxy-core.ts. Allows claude-code pods to make LLM calls without API keys.

Removed files: nats-ipc-handler.ts, nats-llm-proxy.ts, and nats-session-protocol.ts have been replaced by HTTP routes in server-k8s.ts using shared llm-proxy-core.ts and SessionPodManager for session-long pod reuse.

Workspace Operations

Workspace persistence uses git-based providers:

• Local mode (git-local): Bare repos at ~/.ax/repos/{encodedAgentId}. Host handles git clone/pull/commit for file:// URLs.
• K8s mode (git-http): Repos created via HTTP POST to ax-git.{namespace}.svc.cluster.local:8000/repos. Agents clone via HTTP. Git sidecar handles commit/push in k8s pods.

The AX_HOST_URL env var (http://ax-host.{namespace}.svc) is passed to sandbox pods via extraSandboxEnv so they can reach HTTP endpoints. NetworkPolicy allows sandbox pods egress to host on port 8080.

Session Pod Reuse (K8s)

SessionPodManager handles session-long pod reuse via HTTP. Warm pool claiming no longer uses NATS queue groups. The separate warm-pool-client.ts and NATS-based work dispatch have been removed.

Agent Registry

• agent-registry.ts — In-memory/file-based agent registry (registry.json) for single-server deployments.
• agent-registry-db.ts — Database-backed registry for PostgreSQL (k8s). Runs its own Kysely migration (registry_001_agent_registry). Stores agent metadata: id, name, description, status, parent_id, agent_type, capabilities.

Admin API (server-admin.ts)

Admin endpoints for agent management and diagnostics. Protected by admin token (logged at startup for pod retrieval). Includes admin API endpoints for MCP server management under /admin/api/agents/:id/mcp-servers.

Common Tasks

Adding a new HTTP endpoint:

1. Add URL match in handleRequest() in server-local.ts
2. Create handler function (follow existing patterns)
3. Return JSON with Content-Length header
4. For streaming endpoints, use SSE format (handleEvents pattern)

Adding a new IPC action handler:

1. Add Zod schema in src/ipc-schemas.ts (.strict())
2. Add handler in domain module (e.g., createLLMHandlers)
3. Register tool in src/agent/ipc-tools.ts AND src/agent/mcp-server.ts
4. Update tool count in tests/sandbox-isolation.test.ts
5. If handler emits events, add event types to event bus

Adding streaming events:

1. Emit via eventBus.emit() in completion/agent/provider paths
2. Use dot-namespaced event type (e.g., llm.token_delta)
3. Include requestId for per-request filtering
4. Never include secrets in event.data

Gotchas

• Dequeue by ID, not FIFO: FIFO dequeue causes session ID mismatches and empty canary tokens.
• Consumed response body in proxy retry: After reading body for 401 check, original response is consumed. Retry must use new response.
• Pino uses underscore keys: logger.info('server_listening') not 'server listening'.
• identity_write / user_write / identity_propose skip global taint gate: These do soft queuing in their handlers.
• Collect stdout/stderr in parallel: Sequential collection deadlocks when one pipe buffer fills.
• Channel deduplication: TTL-based processedMessages map keyed by channelName:messageId.
• Extract image_data blocks immediately: image_data blocks are transient -- must never be stored in conversation store. extractImageDataBlocks() is the single conversion point.
• Event emission is synchronous: Emits never await or throw.
• Plugin integrity verified on startup: verifyPluginIntegrity() called before PluginHost registers.
• Zombie counter in delegation: Always wrap activeDelegations mutation in try/finally.
• Request-scoped _sessionId for images: Stripped before schema validation.
• SSE keepalive: handleEvents sends :keepalive\n\n every 15s. Watch for keepalive timer leaks on early connection close.
• Port flag support: --port <number> listens on TCP instead of Unix socket.
• Concurrent IPC calls: Shared Unix socket can misroute responses if multiple IPC calls are in flight. Fixed by correlating request/response IDs.
• proxy.sock race: Must await IPC server listen before spawning agent, otherwise agent connects before socket is ready.
• agent_response timeout: Handle timeout gracefully without crashing the host process.
• Container three-phase orchestration: Docker/Apple sandboxes use provision (network on) → run (network off) → cleanup (network on). Do not assume network is available during the run phase.
• K8s IPC is now HTTP-based: nats-ipc-handler.ts, nats-llm-proxy.ts, and nats-session-protocol.ts have been removed. K8s pods use HttpIPCClient → POST /internal/ipc HTTP route. NATS is for event bus only; work dispatch uses HTTP via SessionPodManager.
• Deleted files: nats-sandbox-dispatch.ts, agent-runtime-process.ts, local-sandbox-dispatch.ts, nats-ipc-handler.ts, nats-llm-proxy.ts, nats-session-protocol.ts, server.ts, host-process.ts are all removed. Agent-side workspace files (workspace-cli.ts, workspace-release.ts, workspace.ts) have also been deleted — workspace operations now go through IPC handlers.
• Error redaction in streams: Streaming responses must redact internal errors before sending to client.
• Web proxy per completion: When config.web_proxy is enabled, a web proxy instance starts per completion (Unix socket for container sandboxes, TCP for subprocess). Cleanup happens in the completion finally block.
• K8s web proxy: In k8s mode, web proxy runs as a TCP server (port 3128, bound to 0.0.0.0) in the host process. Sandbox pods connect via k8s Service (ax-web-proxy.{namespace}.svc:3128), passed as AX_WEB_PROXY_URL. Requires: (1) config.web_proxy: true in config, (2) webProxy.enabled: true in Helm values, (3) host network policy allowing ingress on port 3128 from execution plane. The web_proxy field must be in the Zod schema in config.ts.
• K8s env var naming: Never use env var names that K8s service discovery auto-generates. The ax-web-proxy Service generates AX_WEB_PROXY_PORT=tcp://IP:PORT in all pods. Our custom env var is AX_PROXY_LISTEN_PORT to avoid collision.
• Session pod payload propagation: Per-request env vars must be in the pod spec (sandboxConfig.extraEnv). The runner's parseStdinPayload() must extract and applyPayload() must set them in process.env. SessionPodManager handles session-long pod reuse via HTTP.
• AX_HOST_URL env var: Passed to sandbox pods for HTTP endpoints. Points to http://ax-host.{namespace}.svc. NetworkPolicy must allow sandbox→host on port 8080.
• Workspace is git-based: git-local (file:// bare repos) for local mode, git-http (HTTP repos via ax-git Service) for k8s. Old GCS/local/none workspace providers and staging store have been removed.
• Status SSE events: server-completions.ts emits type: 'status' events via EventBus with {operation, phase, message} data. server-request-handlers.ts validates all three fields are strings before forwarding as SSE named events. Invalid payloads are logged and dropped.
• MCP fast path runs in host process: inprocess.ts has no sandbox isolation — tools execute directly. Only safe for trusted MCP providers. FAST_PATH_LIMITS enforces per-turn resource limits.
• MCP provider category: mcp category in provider-map.ts with none (no-op) and database implementations. The database provider requires { database, credentials } deps passed via registry.ts (not the generic loadProvider path). Admin API endpoints under /admin/api/agents/:id/mcp-servers. Unified tool discovery via McpConnectionManager — tools from plugins, database MCP servers, and default provider resolved in one pass.
• credential_request handler in skills.ts: The credential_request IPC handler lives in src/host/ipc-handlers/skills.ts, not in its own file.