self-knowledge

name	self-knowledge
description	Use when the user asks what Fermix is, what it can do, or how its agent loop, providers, channels, jobs, memory, sandbox, built-ins, skills, plugins, and config/setup surfaces fit together.
allowed_tools	[]

Fermix self-knowledge

Fermix is an Elixir/OTP agent daemon. fermix run or the OS service starts one BEAM VM; the CLI talks to it over $FERMIX_HOME/daemon.sock. Channel messages enter the gateway, run one agent turn, execute capabilities through the registry, and reply through the originating channel.

Runtime shape

• Main Agent caches runtime context; channel messages go through the gateway, not directly to Main Agent.
• Gateway serializes one FIFO turn per conversation; different conversations run concurrently.
• Turn flow: read history -> build prompt -> model/tool loop -> reply -> commit messages -> background memory review -> optional auto-compaction.
• Main turns and new scheduled jobs default to 100 iterations. Regular subagents fan out wide (hard-max 10 tasks, ≤8 concurrent — default 4; 100 iterations each). /ultra is a run-mode of the normal turn (not a separate orchestrator): it tags the turn run_profile: :ultra, unlocking wider subagents caps (subagent_mode: :ultra → up to ~50 narrow probes, ≤12 concurrent, reduced per-probe depth) plus an exhaustive-mode prompt addendum driving both breadth (many narrow probes) and best-of-N depth (independent subagents on the same hard sub-problem, keep the best-supported answer). Its workers nest under the parent trace like regular subagents; they stay brief by instruction, not a low ceiling. Repeated identical tool calls trip the loop detector. These knobs are system-side config (:subagents, :ultra, :iteration_limits), not a config.toml surface.
• Sub-agents and unpinned scheduled jobs inherit the main model by default; [fermix_core.routing] subagent_*/cron_* (model/provider/reasoning_effort) can run them on a smaller model or lower thinking level — set via config.toml, the setup page (sub-agent only, on the primary provider's pane), or the model_routing_config tool; a model set without an explicit provider runs on the primary provider (a cross-provider worker needs an explicit subagent_provider/cron_provider). An explicit provider paired with a model the catalog knows under a different provider (e.g. an Ollama slug under openrouter) is a mis-pairing: it's rejected at write time (the tool validates the merged routing) and fails loud at spawn — so only a hand-edited config.toml can hold one, and even then it errors clearly instead of silently. The main agent can also pass a one-shot model to the subagents tool for a single call. It never changes its own model.

Providers

Provider selection is primary-flag driven: each [fermix_core.providers.<name>] block carries primary = true|false, exactly one provider is primary, and every other configured provider is an automatic fallback (deterministic catalog order: openai_codex, openai, anthropic, xai, openrouter, mistral, ollama — Ollama deliberately last: a local model is the last-resort hop). Setup marks a newly configured provider primary by default and clears the old flag (settings/credentials are kept); the web setup provider page renders per-provider cards (status: Primary / Fallback / Not configured) as the provider selector — picking a card loads that provider into the "Configuring …" form, and saving it makes it the primary (needs a daemon restart); a configured, non-primary card also shows a "Set primary" button that flips the primary flag without re-entering credentials. Nothing is disabled, so any provider can be selected and set up. A Realtime API key reuses the OpenAI provider key slot but does not, on its own, promote OpenAI to primary — only a real provider credential or an explicit primary choice does. CLI OAuth login never changes primary. The legacy [fermix_core.agent] provider key is read-only migration input (used only when no primary flag exists; setup stops writing it). Routes are snapshotted at boot for main turns (restart to apply changes); scheduled jobs without an explicit provider resolve the chain at execution time — that asymmetry is intentional. Failover happens only on the turn's initial model call for transient errors (timeout/transport/5xx/rate-limit/quota, or a residual OAuth auth failure after the adapter's own refresh+retry); API-key auth failures, context-length, tool errors, mid-tool-loop (continue) errors, and mid-stream failures after visible streamed content never fail over. Before failover, the model call gets a bounded same-provider retry with short exponential backoff on transient infrastructure errors (the connection_unavailable pool-checkout / wake-from-sleep race, transport timeout/close/network, provider 5xx), so a brief flake self-heals on the same provider instead of surfacing or burning a failover hop. The retry budget is spent before any failover hop, on every route — the next route is a different model and the loop pins the winning route for the rest of the tool loop, so hopping on the first transient would silently re-target the whole turn onto a weaker model under a status: ok. connection_unavailable is network-wide so it retries-same and never fails over at all; scheduled jobs opt out of this inner retry entirely (so the two retry loops never stack and a slow provider :timeout cannot overrun the job's configured timeout) and keep their own coarser deadline-bounded backoff, which covers only the wake-from-sleep pool-checkout race — a cron run therefore still fails over on the first transient of any other kind. Separately, a continuation call (mid tool loop) that fails transiently is re-issued in place on the same route with a short bounded backoff, on every surface including scheduled runs; it replays no tools and never switches provider. The continuation's retryable classes are explicit: a transport timeout the adapter MEASURED as pre-response (zero response chunks seen — a connect-phase or first-byte stall; only chunk-counting adapters like Codex can prove this, so a buffered adapter's timeout never qualifies), a transport cut or network error, and a provider-declared unavailability/overload — and only when the failed attempt itself streamed nothing user-visible (a retry after visible content would duplicate it). Unmeasured timeouts, connection_unavailable, rate limits, and everything else still surface on the first failure. A genuine rate-limit/quota whose body carries a reset time surfaces a friendly "usage limit — try again in ~N min" reply. Failovers emit [:fermix, :provider, :failover] telemetry, fermix doctor lists configured fallbacks, and agent status exposes primary_provider/fallback_providers. Values: openai, openai_codex, anthropic, xai, openrouter, mistral, ollama. All seven run turns; the registry behind them is FermixCore.Providers.Descriptor (one declarative entry per provider — labels, auth modes, setup fields, config-key allowlists). OpenRouter ([fermix_core.providers.openrouter]: api_key via OPENROUTER_API_KEY, optional base_url, default_model, primary) rides the Chat Completions wire with vendor-prefixed model ids (anthropic/claude-sonnet-4.6, dots not dashes) and static attribution headers (HTTP-Referer: https://fermix.sh, X-Title: Fermix); no reasoning-effort field is sent (server default). Mistral ([fermix_core.providers.mistral]: api_key via MISTRAL_API_KEY, optional base_url, default_model, primary) also rides the Chat Completions wire (three rolling -latest tiers — mistral-large-latest/mistral-medium-latest/mistral-small-latest) with a plain Bearer key and no attribution headers; like OpenRouter it sends no reasoning-effort field. Mistral's strict validator rejects an assistant turn carrying empty-string content alongside tool_calls, so the shared Chat Completions adapter omits the content key whenever tool_calls are present (a wire shape valid on every Chat Completions provider). Ollama ([fermix_core.providers.ollama]: base_url — presence is what marks it configured, env OLLAMA_BASE_URL; default_model; primary) is keyless (auth_mode :none internally — no Authorization header, no secret) against the local OpenAI-compat endpoint (http://localhost:11434/v1 default; remote/Tailscale hosts work) with a 300s receive timeout for slow local inference; catalog windows are model capability while the server may serve far less and truncate silently, so the doctor probe also POSTs the native /api/show and fails loud when the served num_ctx undercuts the catalog window (fix via OLLAMA_CONTEXT_LENGTH or a Modelfile num_ctx); a 404 probe means the model isn't pulled (ollama pull <model>). The web setup pane detects the Ollama server with a single probe — the configured URL either serves GET /api/tags or it doesn't: a reachable server lists only the locally installed models in the model picker; an unreachable one shows the error with ollama serve/install guidance and a free-form model input. The OpenRouter pane fetches the live upstream catalog (GET /api/v1/models, tool-capable models only, newest first) so all current models are selectable; on fetch failure it shows the error and a free-form input (FermixCore.Providers.ModelListing; static catalog stays authoritative for defaults and context windows). The "Model behavior" panel (reasoning effort / Codex fast) is hidden for providers without behavior knobs (OpenRouter, Mistral, Ollama). Unknown TOML keys in any provider block (and an unknown [fermix_core.agent] provider) fail loud at config load instead of being silently dropped. SpaceXAI Grok supports two auth modes via [fermix_core.providers.xai] auth_mode = "api_key" | "oauth": bearer API key (api_key, XAI_API_KEY), or Grok Build subscription OAuth (loopback PKCE, profile xai_oauth; connect with fermix auth login --provider xai; a 403 means the Grok plan lacks API access — not a stale token). Both ride the OpenAI Responses wire shape with efforts none|low|medium|high (some Grok models reject effort and get it omitted; slash-containing enum values are stripped from tool schemas). Anthropic supports two auth modes via [fermix_core.providers.anthropic] auth_mode = "api_key" | "oauth": API key (api_key), or Claude subscription OAuth (profile anthropic_oauth in auth.json; connect with fermix auth login --provider anthropic using --setup-token, --import-claude-code, or CLAUDE_CODE_OAUTH_TOKEN). OAuth requests emulate Claude Code (identity headers + system block, mcp_-prefixed tool names) and auto-refresh with one 401 retry. All Anthropic requests send prompt-cache breakpoints, request adaptive thinking on models that support it (the model decides when and how much to deliberate; reasoning_effort calibrates it; older models like Haiku 4.5 get no thinking param), and cap non-streaming output at max_tokens 16384 — sized so thinking plus the visible answer fit the buffered receive window. Per-provider model/effort: [fermix_core.providers.<name>] (default_model all providers; reasoning_effort only for effort-capable providers — OpenAI/Codex/Anthropic/SpaceXAI; Codex-only fast). OpenRouter/Mistral/Ollama blocks reject a reasoning_effort key at config load, routing-level effort overlays skip their routes, and their telemetry reports reasoning_effort: nil. Reasoning effort is one canonical vocabulary (FermixCore.Providers.ReasoningEffort: none|low|medium|high|xhigh|max) with per-provider subsets, mapped to each provider's wire field (OpenAI/Codex/SpaceXAI reasoning.effort; Anthropic output_config.effort). Anthropic has no none (floor low, API default high); a level above a provider's ceiling clamps. On top of the provider subset a model can carry its own ceiling in the catalog: max is a GPT-5.6-family capability, so the older OpenAI/Codex models (gpt-5.5, gpt-5.4, gpt-5.4-mini) top out at xhigh and an over-reaching config self-heals down to that model ceiling at route resolution instead of 400-ing at the provider. Models with no catalog ceiling pass through untouched (Anthropic's per-model nuance is left to the provider's own 400). Both the CLI wizard and web setup offer effort for the effort-capable providers only, and list only the levels the selected model actually accepts. auth_mode (api_key/oauth) round-trips through config for SpaceXAI and Anthropic. The web setup provider page has an API-key vs OAuth picker per provider (SpaceXAI = loopback "Connect Grok" like Codex; Anthropic = paste a claude setup-token or import a Claude Code login). A stored token is inert until auth_mode = "oauth", so connecting in the web AND fermix auth login --provider xai|anthropic both set auth_mode = oauth in config (and fermix auth logout reverts to api_key); the change needs a daemon restart. The web setup Media tab also exposes an editable OpenAI/SpaceXAI key field inline next to the image-backend picker (same openai_api_key/xai_api_key provider secret, the same key those providers use for chat): it shows as already-configured when a key is stored — blank keeps the current key, a pasted value replaces it — so the generate_image key can be set without opening the provider's full setup form.

Provider/channel HTTP uses shared FermixCore.Finch. Idle keep-alive connections older than 15s are discarded at checkout. Codex retries :closed once only when it happens before response data; mid-response :closed and :timeout are not retried at the HTTP layer. A Codex 200 whose SSE stream delivered no output at all is never read as an empty answer, and the two undelivered facts stay distinct: a stream that DECLARED its failure (response.failed/response.incomplete/error with nothing generated) becomes an API-classified error — overload/server_error text classifies as provider-unavailable (retryable, failover-eligible) and the server's own sentence is quoted verbatim in the user-facing reply, so a provider-side outage never reads as a Fermix defect — while a stream cut with no declared reason is classified as a transport close (retryable on the same route, failover-eligible) with the reason in the log and the trace. A stream that delivered output but never said it finished still returns that output (with a warning), because output items accumulate independently of the terminal event and discarding them would throw away a usable answer — and on a continuation, which has no retry or failover, it would kill the turn outright. A :timeout the adapter measured as pre-response (zero chunks) is retried one level up, at the agent loop's continuation seam (see the provider failover/retry notes above); a mid-stream or unmeasured :timeout is never retried.

Image input: an inbound image is forwarded to the model as a content block on vision-capable providers (Anthropic, the OpenAI Responses/Codex wire, and Grok); OpenRouter and Ollama are model-dependent (text-only local models are gated off). Each provider encodes the image at its own edge from one neutral content part, and the text-only request shape is unchanged, so prompt caching is unaffected. Tool RESULTS can also carry images, not just inbound messages: a tool that returns an image (e.g. the browser screenshot action) hands the bytes back as an image content part the model actually sees, encoded through the same per-provider edge (an Anthropic tool_result content array, or a placeholder tool message plus a following user image turn on the OpenAI-shaped wires). A continuation carrying an image to a non-vision route fails loud, never silently drops.

Built-in capabilities

Built-ins seed into the single capability registry at boot; outbound MCP tools register as mcp_<server>_<tool>. The runtime prompt's ## Built-in Capability Catalog is authoritative; use tool_help for one tool's schema/failure modes.

• Files & search: file_read, file_write, file_edit, glob_search, content_search
• Shell & git: shell, git_read, git_write (no push)
• Web: web_search (static facts/no URL), web_fetch (one known server-rendered URL — returns its readable text, or the JSON verbatim when the endpoint serves JSON), browser (JS/dynamic/interactive; fill replaces, type appends, submit submits; receipts are immediate; verify async changes with wait/get; get field=rect returns a selector match's viewport box in the same CSS space click_coords clicks in — the deterministic route onto a board/map/canvas the snapshot lists no elements for; screenshot returns the captured page as an image you can SEE (device pixels — its device_pixel_ratio converts to click_coords space); a managed browser caps live tabs, so each open past the cap auto-closes the oldest non-active tab — don't assume a long-idle tab is still open; see browser-guidance). Never shell-scrape JS sites.
• Media: generate_image — create or edit a raster image from a text prompt; the result is written under the sandbox media/ floor and sent to the current chat automatically (file-only when no channel is available, e.g. a subagent/job). operation is generate (default) or edit; an edit's input_image is a sandbox path or inbound:last (the image just sent in this chat). Optional mask (transparent regions only — OpenAI backend only), size, model. The operator's config ([fermix_core.tools.generate_image] backend) picks the provider — openai, xai, google, or openai_codex; edit and mask are gated against the chosen backend's capabilities and rejected loudly when unsupported, never silently dropped. Reuses the OpenAI/SpaceXAI chat key, or GEMINI_API_KEY for Google. The openai_codex backend is different: it needs no API key and generates gpt-image-2 through the ChatGPT-subscription Codex OAuth connection (billed to the subscription, not the platform API), via the built-in image tool on the Codex responses endpoint — an experimental, undocumented surface gated to ChatGPT auth (a plan that does not entitle it returns auth_failed). It supports generate + edit (no mask); a router_model config key names the GPT-5.x model that carries the image tool. Not for diagrams/charts/code assets or live data; voice sessions cannot use it (no chat channel to deliver to).
• Memory: memory_store, memory_recall, memory_sources_list
• Jobs: schedule_job, update_job, list_jobs, pause_job, resume_job, remove_job, run_job_now, list_job_runs, get_job_run
• Skills & delegation: skill_view, skill_run, skill_list, skill_create, skill_reload, subagents (bounded temp sub-agents)
• Coding harness (enabled by default but no harness tool is offered at all until the owner sets approved in Setup → Coding Agents; each run tool then advertises only when its vendor CLI was detected at boot, and when default_vendor is set only that vendor's run tool is advertised): codex_run, claude_code_run (delegate a repo coding task to the operator's own Codex / Claude Code CLI, run inside that repo), list_coding_runs, get_coding_run, cancel_coding_run (inspect and cancel runs). The cloud rail (codex_cloud_run + stop_tracking_coding_run) is off in this release ([fermix_core.harness] cloud_enabled = true to enable).
• Meta: tool_help, send_attachment (send a local sandbox file out through the active channel — outbound only; inbound images are materialized by the gateway, not via a tool; URLs rejected), react (acknowledge the user's current message with a single emoji reaction instead of a text reply — offered on channels that support reactions: Telegram, Discord, Slack, WhatsApp, Signal; absent on the CLI; ends the turn with no text bubble), model_routing_config (set the sub-agent model in [fermix_core.routing]: subagent_model/subagent_provider/subagent_reasoning_effort)
• Tool-schema deferral (M10): default-on — [fermix_core.tools.tool_search] enabled absent means true; enabled = false is the kill switch. When on, plugin/MCP tool schemas leave the provider wire (names stay listed under ## Plugins) and three bridges register: tool_search (BM25 over the deferred catalog), tool_describe (one tool's full schema on demand), tool_call (invoke a deferred tool; the loop unwraps it so traces/policy see the real tool name — direct calls by name also work). Off = bridges absent, all schemas inline.

Built-ins need no API keys except alternate backends/integrations; default web_search is DuckDuckGo. If a configured non-DuckDuckGo backend (Brave/Exa/Tavily/Firecrawl/etc.) hard-errors (auth, credits/HTTP 402, transport), web_search degrades once to keyless DuckDuckGo — loudly (a warning log + degraded/primary_backend/fallback_reason in the trace), not silently — so the broken backend stays visible. Empty results do not trigger the degrade.

Skills

Skills are SKILL.md instruction packages, not provider-visible tools. skill_view loads a body; skill_run delegates as a sub-agent (recursion cap 4); skill_list enumerates. Bundled priv/skills and local ~/.fermix/skills load at operator trust; plugin skills load at the guest trust level (capability-restricted), independent of the caller's gateway trust. allowed_tools narrows the trust default ([] = none, absent = trust default). skill_create writes to ~/.fermix/skills; skill_reload re-scans the skill directories and refreshes the running agent in place (no restart) after a SKILL.md is created or edited on disk, reporting added/removed/changed names and load errors. CLI: fermix skills list|view NAME|reload.

Memory

Durable memory is SQLite at ~/.fermix/memory.db; ETS is cache. memory_recall does key lookup or FTS5 search over memories/history with scope (current|owner|all) and source (memories|history|all). scope is a request, not an authorization: the effective scope is resolved from the caller and the argument can only narrow within it — a scheduled run always reads its own job memory and a guest always reads the conversation it is in, whatever scope is named. Facts carry one of six categories on a general-assistant spine: identity, preference, interest, goal (about the user) and context, directive (about the work). Background review ("dreaming") consolidates rows on a ~24h gate and rebuilds two prompt-injected profiles — USER.md (the four user categories) and MEMORY.md (context/directive) — replacing stale facts rather than appending. CLI: fermix memory review --now, fermix memory restore ID.

Jobs

schedule_job creates durable work without running it now. Schedule forms: interval (every N minutes|hours|days), one ISO8601 datetime (once), or 5-field cron; free-form English (e.g. "daily at 8am") is rejected — use 0 8 * * *. Each cron field supports *, single values, comma lists (1,15), ranges (9-17), and steps (*/15, 8-18/4); weekday 7 and 0 both mean Sunday; out-of-range or malformed fields are rejected at creation. Cron fires in the job's timezone (DST-aware; an unknown zone is rejected at creation). Runs are isolated bounded agent loops and cannot see the creating chat, so include needed facts in task text. If the daemon was down across a recurring job's fire time, a due time older than the freshness window ([fermix_core.jobs] run_freshness_window_seconds, default 3600) is skipped rather than fired late at the wrong wall-clock — the schedule just advances to the next future occurrence (logged). A one-off once run has no next occurrence, so it runs late instead of being dropped. A schedule expression or timezone that no longer parses (e.g. a corrupted row) is terminal: the job moves to a disabled state with the parse error in last_error and stops being retried — it must be fixed with update_job and then resumed, distinct from a user pause. Concurrent scheduled runs are capped (a small fixed ceiling); while the cap is full a due job stays scheduled and is claimed by a later tick as a slot frees, so a burst of simultaneously-due jobs never fans out without bound. A transient failure on the whole-loop retry is only retried before any tool has executed — once a tool has run, a mid-run connection loss fails the run loudly rather than replaying the tool's side effects (one narrow exception: a continuation call that failed transiently — a pre-response timeout, a transport cut, or a provider-declared overload — with nothing user-visible emitted is re-issued in place by the agent loop with short bounded backoff; nothing replays). A run that fires just as the host wakes can hit a not-yet-ready network: the runner classifies a pool-checkout connection_unavailable failure as transient infrastructure (not a provider-failover case) and re-runs the whole loop with bounded exponential backoff, and — when [fermix_core.jobs] network_readiness_enabled (default true) is on — first waits on a short, bounded TCP readiness probe to the primary route's host before the first model call. expires_at makes a temporary job; delivery_mode is none|origin|channel|local. timeout_seconds caps each run's wall clock (absent = the 30-minute daemon default) and inactivity_timeout_seconds arms a watchdog that fails a run whose provider/tool loop stops making progress (absent = unarmed); both are set at creation only — update_job cannot edit them — and get_job_run's config snapshot echoes the values a run actually executed under. allowed_tools narrows the run to a subset of the caller's currently-visible tools (unknown names rejected); the model can never widen the run's capability policy past the creator's trust. Operator-created scheduled runs can delegate in parallel via subagents (regular caps — 10 tasks / 8 concurrent), and each worker's surface is the intersection of the delegation baseline and the run's own ceiling: a job confined by allowed_tools/capability_policy/skill_name spawns workers confined the same way, never wider. Guest-created runs never see subagents (it is policy class external_api, which the guest surface excludes), and subagents is only advertised to a run where it can actually execute. Every job is stamped at creation with its creator's own trust (operator or guest) — a context that carries no trust cannot create a job at all, so a job never inherits a trust the creator lacked. skill_name binds the run to an existing skill (rejected at creation if unknown): the run then executes inside that skill's confinement — the skill's allowed_tools and policy are intersected with the job's, never widened, and a guest job naming a skill whose policy grants nothing under guest trust fails loud rather than running unconfined. Optional provider + model pin which provider/model the job's runs use; they are both-or-neither (set both or neither — a pin without its pair is rejected), provider must be a known/configured provider (validated at creation against the same catalog the runner enforces), and model is a free-form provider-specific id. Omit both to use the default cron route ([fermix_core.routing] cron_*, else the primary/fallback chain resolved at run time). update_job edits a job in place — task, schedule, description, skill_name rebinding, provider/model route pin, and delivery (delivery_mode/delivery_target); omitted fields are left unchanged (delivery is never silently retargeted to a config default, and an omitted provider/model keeps the current pin), and switching delivery to none/local clears the target. A clear_route_pin boolean un-pins the job's provider/model back to default routing; it is mutually exclusive with provider/model (set those to re-pin instead) and combining them is rejected. list_jobs payloads surface task_prompt (the job's current instructions), skill_name, provider, model, delivery_mode, and delivery_target so the instructions, binding, pinned route, and destination are readable without reaching into the database. run_job_now fires a job immediately, out of band, through the same isolated runner (the run is tagged trigger: "manual") and leaves the timed cadence untouched — use it to test a job or satisfy an on-demand request; it refuses when the job is paused/disabled/expired or already mid-run. list_job_runs reads a job's execution history (status/trigger/timing/outcome, newest first, optional status filter) and get_job_run reads one run in full (task_prompt the run actually executed — captured in its config snapshot, so it reflects the instructions at run time rather than the job's current ones; plus prompt snapshot, token usage, final response, error) — use these to confirm a job is actually firing and inspect what its runs produced.

Channels & access control

Channels: Telegram, WhatsApp, Slack, Discord, Signal (text + media), plus local cli and daemon. Remote channels refuse to start without owner_user_id or allowed_*_ids. Inbound images on media-capable channels are downloaded at the gateway (sibling to audio transcription) and passed to the model as image content — the agent sees the picture, not a placeholder; audio attachments are transcribed to text first on all five remote channels — Telegram (voice notes, audio files, and video notes), WhatsApp, Slack, Discord, and Signal. When a voice note also carries a caption, both are delivered: the caption first, then the transcript under a [voice note transcript] delimiter. Transcription needs a configured backend, selectable in setup ([fermix_core.transcription] backend = openai | xai | deepgram; CLI flags --transcription-backend/--transcription-model/--transcription-api-key, plus a Transcription card on the web setup and a transcription row in fermix doctor). Each backend has its own optional key slot, always settable in the Transcription card (and --transcription-api-key stores under the selected backend's slot): openai (default, gpt-4o-mini-transcribe) and xai (SpaceXAI, Grok STT — modelless, so it has no model to pick and the card hides that field) take a transcription key that OVERRIDES the reused chat-provider key, or reuse that chat key if none is set. SpaceXAI STT REQUIRES an API key — the Grok subscription OAuth token does not work for /v1/stt — so paste one when the SpaceXAI provider is on OAuth. deepgram (nova-3) has no chat provider to reuse, so its deepgram_api_key is required. All three keys keychain as @keyring. model is a single shared key, so setup snaps it to the chosen backend's default on a backend switch (an unknown backend or non-positive max_file_mb fails config load loudly). When it isn't configured, the file is over the size cap, or the provider errors, the sender gets a specific reply instead of a silent drop (not configured → run fermix setup; too large → the size-cap limit; other failures → transcription failed, try again) and no turn is scheduled. An image whose resolved model can't accept vision fails loud rather than dropping the image silently. Multi-image messages are coalesced into one turn so the agent sees every image together — Telegram albums (separate updates sharing a media_group_id) and WhatsApp's separate per-image webhook messages are both buffered by a short debounce and merged; Discord/Slack/Signal already deliver all attachments in one message. An inbound message with no text AND no media (a sticker, poll, or blank — a captionless image still counts as actionable) is answered at the gateway with a brief "looks empty" note and never schedules a turn, so the queue stays free and no model is called; an unauthorized sender is dropped one step earlier and gets no reply at all. If a model ever returns an empty completion, the turn replies an honest "try again" instead of a blank message, and that empty reply is never committed — so it can't poison later turns. The conversation store independently refuses to persist or replay any content-less turn of any role.

Live streaming (on by default — "block"; set streaming = "off" per channel to disable): [fermix_channels.<name>] streaming = "off" | "draft" | "block". "draft" shows the reply as one draft message edited in place (~1/s, ≥30 chars before the draft opens) and sealed to the final authoritative text; /stop deletes the draft; needs a draft-capable channel (Telegram today; fermix doctor warns otherwise). "block" sends each completed model "thought" as its own ordinary message (semantic boundary = completed output item; 800–1200-char paragraph chunking only as the long-text fallback, 1 s idle flush), including the model's 💭 reasoning-summary headings as separate one-line messages — works on every channel, and pre-tool commentary lands as its own message. Both need a streaming provider (Codex today — others simply deliver normally). Only real channel turns stream — background/CLI/cron runs never do. Stream telemetry: [:fermix, :channel, :stream] with phase open/edit/block/seal/discard and the turn's session_id.

Gateway trust: operator (owner_user_id or any local caller) gets full surface; guest gets read-only chat: no skills, MCP, exec, or network. allowed_*_ids authorizes chat only, not operator promotion. Read-only is a bound on what a guest may do, not on whose data comes back, so the surface is narrowed on a second axis too: capabilities that return the owner's own data — workspace reads (file_read, glob_search, content_search, git_read), scheduled-job reads (list_jobs, list_job_runs, get_job_run), and memory_sources_list — are never in a guest's surface, in the prompt, on the wire, or dispatchable by name. Tool discovery (tool_search, tool_describe, tool_help) is filtered by the same ceiling, so a guest cannot read the schema of a tool it cannot call.

Slash commands are pre-agent: /help, /whoami for any authorized sender. Operator-only: /compact, /new (/clear), /sandbox (/grant, /revoke, /confirm), /soul, /stop, /pause, /resume, /background (/bg), /tasks, /ultra. /pause and /resume control computer use: /pause hands the cursor and keyboard back to you mid-task (the session stays alive and resumable — unlike /stop, which tears it down), /resume lets it continue. Guests can be opted into owner-only commands via command_allowlist — but the sandbox mutation subcommands (/grant, /revoke, /confirm) are strictly operator-only and never reachable through the allowlist (a guest can chat and run /new but can never grant directory access or apply a confirmation). Auto-compaction runs after reply at threshold (default 0.85); /compact forces it.

/stop kills active turns and clears pending work; the stopped turn neither delivers nor commits a reply. Its user message was already persisted at turn start, so the gateway appends a short assistant marker after it ("stopped before I finished … context only") — the discarded request stays in history/memory but the next turn does not replay and answer it. The marker is only added when the conversation's last stored message is that orphaned user turn (no double-marking, role alternation preserved).

/soul curates the persona file (SOUL.md), owner-only and never autonomous. /soul alone reports the current revision; /soul review [instruction] [--with-context] drafts an edit — no instruction means a subtle, voice-preserving review held to a small change budget (declines if nothing warrants a change); an instruction is an explicit, unbounded ask; --with-context folds a hard-bounded window of the owner's own recent messages into the draft as evidence (guest turns are filtered out). The draft is one bounded provider call that advertises no tools and never writes; it returns a diff plus rationale, and prompt-injection markers detected in source memory are surfaced as a warning on that diff. Applying needs a second explicit step — /soul apply TOKEN (the token expires after a few minutes, enough time to read the diff; /soul diff TOKEN re-previews). /soul history lists revisions, /soul revert N rolls back to revision N, /soul reset restores the shipped default; every write (including reverts and resets) is versioned in the resource registry and itself revertable. The draft run emits [:fermix, :soul_curation, :run_start|:run_complete|:run_error] under its own minted session_id so the bounded provider call reassembles into one trace.

Sandbox

Filesystem/exec built-ins (shell, file_*, search, git_*, send_attachment) go through the sandbox. Modes: strict (workspace + granted roots), standard (default: workspace + the launch/request cwd when it is under your OS home $HOME + granted roots — so the agent works where you ran fermix ask, with no hardcoded project-folder list), open (all of $HOME except protected/blocked). The request cwd is admitted only for a trusted local (operator) turn; remote channels fall back to workspace + grants. Protected paths key off the OS home (~/.ssh, ~/.aws, ~/.gnupg, …) plus Fermix internals and OS roots, and are denied in every mode even inside a granted root (protected always wins); catastrophic commands are denied in every mode too. Child env is minimal; secrets come from host env, keyring, or source = "command" helpers. Roots live in [sandbox] config as workspace_root + allowed_roots, managed via fermix grant|revoke path PATH, fermix sandbox ..., or /sandbox (web setup has no roots field — roots are CLI/config/chat only). Command profiles expose extra local commands as tools. In-chat approval loop: when a filesystem op is denied for being outside the roots and the task genuinely needs that directory, the agent (in an attended operator turn only) calls the request_directory_access tool — the owner sees the exact canonical path, the reason, and the config diff, and approves with /confirm TOKEN (single-use, origin-bound to that conversation, owner-only, 60s expiry). On Telegram and Discord the prompt also carries a one-tap Approve button whose payload holds the token privately (Telegram inline-button callback, Discord message component) — tapping it funnels through the same operator-only, single-use /confirm path; the Discord app must be configured to receive component interactions over the gateway. Because that button delivers the token, in a shared/group chat the raw /confirm TOKEN line is dropped from the group-visible text (it would otherwise expose the token to every member) and the owner approves via the button (or /confirm in a DM or the CLI); a non-owner who taps is refused before the token is consumed and (on Discord) sees an ephemeral "not authorized" note. Everywhere the token is NOT delivered privately by a button — a DM, the CLI, or any channel without the one-tap button (Slack, Signal, WhatsApp) — the tap-to-copy /confirm TOKEN stays in the prompt, because it is the only way the owner can confirm. The system refuses to even prompt for a path it would reject ($HOME wholesale, FERMIX_HOME, ~/.ssh, OS roots). On confirm the grant persists to [sandbox] allowed_roots and, on a chat channel, the original request auto-resumes; on the one-shot CLI the owner re-runs it. A guest, cron, or unattended run never gets this approval path.

Plugins

Plugins are connected integrations that own a surface (an API, a vault) and register their own tools; the always-present bundled trio is Gmail, Google Calendar, and Google Drive (Google OAuth). The agent discovers what is live from the ## Plugins section of the runtime prompt — there is no plugin-list tool — and additional plugins install from a signed in-binary catalog over an http or mcp rail. Full detail: skill_view(name: "self-knowledge", file: "plugins").

Voice (macOS, off by default)

The OpenAI Realtime voice companion (FermixPet) is local and off by default ([fermix_core.realtime] enabled=true plus an OpenAI Platform API key); the macOS app ships from the separate tezra-io/fermix-macos repo and connects over $FERMIX_HOME/realtime.sock. Inside a call the screen_share tool (start/stop) watches the operator's screen continuously — changed frames join the session as passive context, so a still screen costs nothing and a frame never makes you speak on its own; acting on what you see still goes through computer_use/browser. It needs computer use enabled+installed (same sidecar and Screen Recording grant), is voice-only (a text turn takes a computer_use screenshot instead), ends with the call, and is switched off with [fermix_core.realtime] screen_share. CLI: fermix voice status. Full detail: skill_view(name: "self-knowledge", file: "voice").

Computer use (experimental, off by default)

Computer use is experimental and off by default (enabled from the setup Plugins page on Apple Silicon macOS and Linux x86_64); computer_use drives the host desktop GUI by screenshot plus mouse/keyboard, its safety access posture is derived 1:1 from [sandbox] mode, and it is operator-only — never delegated to subagents and never started by an unattended run. It refuses a pointer action whose coordinates are plausible on two image grids at once (magnified crop vs full screen) with a typed conversion error instead of guessing which image the model read. Full detail: skill_view(name: "self-knowledge", file: "computer_use").

Coding harness

Repo coding work is delegated to a coding-harness run rather than edited file-by-file: codex_run and claude_code_run launch the operator's own Codex / Claude Code CLI inside a target repo. It is enabled by default ([fermix_core.harness] enabled) but gated on a first-use consent the owner grants once in Setup → Coding Agents (or [fermix_core.harness] approved in config) — there is no in-chat approval prompt, and until it is granted no harness tool is offered at all (run tools and the run-history tools alike) and Fermix simply does the coding itself with its own file/shell tools. Each run tool advertises only when its vendor CLI is detected at boot, and when default_vendor is set only that vendor's run tool is advertised (the other stays dispatchable by name). The cloud rail (codex_cloud_run) is off in this release — [fermix_core.harness] cloud_enabled defaults false. Full detail: skill_view(name: "self-knowledge", file: "coding_harness").

Config & control surfaces

• FERMIX_HOME default ~/.fermix: files config.toml, auth.json, memory.db, daemon.sock, realtime.sock; dirs workspace/, bootstrap/, memory/, skills/, plugins/, browser/, journals/, realtime/, traces/, logs/, grants/.
• Prompt files from setup: bootstrap/main/{IDENTITY,FERMIX,SOUL,REALTIME}.md, memory/main/{USER,MEMORY}.md; default agent id main. These are seeded once, at setup, and are yours after that — the seeder skips any file that already exists and no upgrade rewrites them, so a fresh install gets the improved shipped templates while an existing one keeps its copy. fermix doctor's bootstrap templates row catches that drift by comparing what this install recorded when it was seeded against the template the current build ships (FERMIX.md/SOUL.md/REALTIME.md — variable-free templates only; IDENTITY.md embeds the agent name and is excluded), and warns which shipped templates have moved on so you can diff your file against the current one for missed improvements. It does not inspect your edits — a hand-edited file is fine. Installs seeded before revision tracking report no seed record.
• Personalization (the USER's name, timezone, communication style) is set at setup; timezone defaults to America/New_York (CLI wizard and web setup). The personalization step also collects the assistant's own name — that is identity, not a user preference, so it persists to [fermix_core.agent].name (default fermix), the source of truth that seeds IDENTITY.md (the first block of the system prompt) and is reconciled into its **Name:** line on each daemon boot — so changing the name in config and restarting updates what the agent answers, while a blank/unset name and any other operator edits to IDENTITY.md are left untouched. The current date (UTC, with the configured timezone as a label) is stamped into the system prompt every turn and into scheduled job runs, so the agent knows "today" without running date. Date-only by design — a clock time would bust provider prompt caches every turn; the precise time still comes from date.
• Config sections: [fermix_core.agent], [fermix_core.providers.<name>], [fermix_core.tools.web_search], [fermix_core.tools.tool_search] (enabled — tool-schema deferral), [fermix_core.tools.generate_image] (backend openai/xai/google/openai_codex + model/size; google_api_key for Google; router_model for openai_codex, which needs no key and uses the Codex OAuth connection), [fermix_core.transcription] (backend openai/xai/deepgram + model/max_file_mb; per-backend openai_api_key/xai_api_key/deepgram_api_key, keychained — openai/xai override the reused chat key, deepgram is required), [fermix_core.jobs|memory|realtime|computer_use|plugins|routing|harness] (harness: coding-harness master gate enabled + first-use consent approved + steer default_vendor + cloud rail cloud_enabled, off by default), [fermix_core.oauth.<provider>] (plugin OAuth clients: google, github, notion, x, slack — client_id + client_secret, secret keyring-secured), [fermix_core.plugin_secrets] (per-plugin api_key credentials, keychained as @keyring, keyed by plugin name), [sandbox], [mcp.servers.<name>], [fermix_channels.<name>].
• Secrets: plaintext config, @keyring, or env vars. Saving config auto-stores plaintext secrets to the OS keyring as @keyring when a writer is available (macOS security; account fermix, service fermix:<ENV> for the default profile or fermix:<profile>:<ENV> when [fermix_core] profile names a non-default one — general/unset keeps the bare fermix:<ENV>, so existing installs need no migration); without a writer, unchanged plaintext is kept (with a warning) and only new secrets fail. On macOS every save deletes then re-adds the keychain item with an open ACL (security -A) so the ad-hoc-signed daemon reads it headlessly with no authorization prompt; a legacy item that still triggers a repeating "security wants to use the login keychain" password prompt (written before the open-ACL delete-then-add) self-heals the next time it's written — re-run fermix setup to rewrite all secrets at once and the prompt stops. The optional [fermix_core] profile (default general) namespaces a host's keyring entries so two installs/workspaces (e.g. ~/.fermix and ~/.fermix-dev) don't overwrite each other's secrets. Set it before running fermix setup — secrets are written under the profile in effect at save time, and there is no migration when the profile changes, so flipping it on a populated install orphans the old entries (re-run fermix setup to re-write them under the new coordinate). fermix setup --migrate-secrets migrates plaintext in place. OAuth logins: fermix auth login (Codex), fermix auth login --provider anthropic (Claude subscription), fermix auth login --provider xai (Grok Build); fermix auth status|logout accept --provider.
• Web setup is local-session gated. fermix setup and the configured home dashboard mint a short-lived one-time /setup?t=... launch URL; the durable setup token is never placed in URLs.
• Service: launchd/systemd, user default or --system; control with fermix start|stop|restart. Inspect with fermix status|health|doctor|logs -f|capabilities; restart for config changes. Binary upgrades need a restart too: fermix upgrade (the built-in updater for standalone installs) swaps the cosign-verified binary and restarts + health-checks itself with rollback on failure, but on a package-manager install (e.g. Homebrew) it refuses to touch the managed binary and points at the package manager — and after brew upgrade fermix the daemon keeps running the old version until fermix restart. fermix status and doctor's daemon-socket check warn when the running daemon's version differs from the installed binary and name the restart fix; "upgraded but behavior unchanged" almost always means the daemon was never restarted. The installed unit pins a PATH (the directory fermix was installed into — its sibling cosign on a Homebrew install — plus the standard system + Homebrew bin dirs) so the supervised daemon can shell out to cosign (plugin-signature verification) and brew-installed node/python (MCP runtimes); a bare launchd/systemd PATH omits the Homebrew prefix, which makes plugin installs fail with a misleading signature invalid. The unit is a snapshot of install-time settings, but fermix setup self-heals a drifted unit — when the on-disk unit no longer matches what the current binary would write (e.g. the PATH or template changed across an upgrade), setup rewrites and reloads it instead of just restarting — so re-running setup picks up unit changes; fermix service install is the manual escape hatch.

Observability

Traces are JSONL under ~/.fermix/traces/YYYY-MM-DD/<type>.jsonl: llm_call, tool_exec, agent_event, channel_msg, error, sandbox_event; logs are ~/.fermix/logs/fermix.log. No fermix traces verb. All log output (file and console, crash reports included) passes through a secret-redaction formatter: credential-shaped tokens (OpenAI/Anthropic sk-…, GitHub, Slack, SpaceXAI, Google, Telegram bot tokens, AWS key ids, bearer headers) are replaced with [REDACTED:<vendor>] markers — a marker in the log means the redactor caught a secret, not that data was lost.

llm_call and tool_exec carry session_id: main turns (main-<n>), subagents (random hex linked by parent session events), scheduled jobs (cron_<job>_<ts>), or realtime voice calls (session:<n>). Content capture is off by default; FERMIX_OPIK_ENABLED=1 enables it unless FERMIX_TRACE_CONTENT=0; FERMIX_TRACE_CONTENT=1 captures locally without Opik. Capture on means full-fidelity traces: input/output bodies are attached whole (no 2k truncation, no inspect eliding), and a failed browser action additionally carries the profile's recent console/JS-exception buffer in its error details — the trace is the one place to look when debugging. Capture off keeps traces bounded and body-free. The in-repo fermix_opik exporter (apps/fermix_opik, bundled in dev/prod, inert unless FERMIX_OPIK_ENABLED) exports nested traces and provides mix opik.replay.

Enabling Opik on the daemon: fermix service install snapshots a non-secret env allowlist (FERMIX_OPIK_ENABLED, FERMIX_OPIK_BASE_URL, FERMIX_OPIK_PROJECT, FERMIX_TRACE_CONTENT, plus the FERMIX_HOME baseline) into the launchd/systemd unit — a shell export alone never reaches the daemon, so reinstall after changing it; the API key is never written to the unit. fermix doctor's "opik export" check asks the daemon over the control socket whether the exporter is off, enabled-but-not-bundled, or ready (with the resolved endpoint/project). Realtime voice calls are fully traced: [:fermix, :realtime, :call_start|session_created|session_updated|provider_error|reconnect|call_stop] lifecycle (→ agent_event), the model turn via [:fermix, :provider, :call], and tool calls on the same session_id — so one call reassembles into one Opik trace.

Reference files

This skill's overview stays in this file; deeper per-feature detail loads on demand as a named reference file:

• coding_harness — the coding-harness rails (codex_run/claude_code_run local delegation, codex_cloud_run cloud tracking), authorization, consent, and durable delivery.
• computer_use — the experimental host-desktop control capability: actions, the access/attended-origin safety model, coexistence, permissions, and config.
• plugins — connected integrations: the http and mcp rails, auth/credential errors, the signed catalog, and the operator CLI.
• voice — the OpenAI Realtime macOS voice companion (FermixPet): setup keys, the tezra-io/fermix-macos distribution, and the connection handshake.

Load one with skill_view(name: "self-knowledge", file: "<name>") (e.g. skill_view(name: "self-knowledge", file: "plugins")).