| name | ai-utils |
| description | Always read this skill before editing any file under packages/ai/ or packages/visual-ai/. Covers AI primitives: the agent abstraction, model registry, structured output generation, visual checkers, and element detection used across the platform. |
AI Utils (@autonoma/ai + @autonoma/visual-ai)
The AI primitives are split across two packages:
@autonoma/ai - the sharp-free core: the reusable agent abstraction, model management, structured output generation, and text utilities. Has no dependency on @autonoma/image, so it is safe to load in hosts without the sharp native binary (e.g. the API).@autonoma/visual-ai - the screenshot-driven primitives: visual checkers and point/object detection. Depends on @autonoma/ai and on @autonoma/image (sharp). Import visual/detection symbols from here, everything else from @autonoma/ai.
This skill is the conceptual map. The live model list, package exports, and detailed usage examples are maintained in packages/ai/README.md and packages/visual-ai/README.md - treat those READMEs (and packages/ai/src/registry/model-entries.ts for the models) as the source of truth and update them when behavior changes.
Agent Abstraction (@autonoma/agent-core)
The agent abstraction and compaction now live in the dependency-free @autonoma/agent-core package (deps: ai + zod only), so they can be bundled into the published planner CLI without @autonoma/logger/@sentry/node or the model registry. @autonoma/ai re-exports every symbol unchanged and registers its Sentry logger as the loop's default at import, so backend code keeps importing them from @autonoma/ai exactly as before. A CLI depends on @autonoma/agent-core directly and injects its own Logger (or leaves it silent).
The most important building block. It is the generic, reusable tool-loop agent that every agentic workflow extends - the reviewers, healing, resolution, and diffs agents in packages/diffs are all subclasses. (Note: the test-execution agent in @autonoma/engine is separate and wraps the AI SDK directly; this abstraction is for the non-execution, reasoning-style agents.) It wraps the Vercel AI SDK's ToolLoopAgent and adds typed results, typed tool error handling, and optional message compaction.
Three classes form the core, with a clean immutable-config / per-run-state split:
-
Agent<TGenerationInput, TResult, TLoop> - an immutable factory holding config and dependencies. Each run(input) builds a fresh per-run loop. Subclasses implement two hooks:
buildUserPrompt(input) - async, so pre-loop work (uploading a video, computing a diff) can feed the prompt.createLoop(input) - construct the per-run AgentLoop carrying any state the tools will read.
-
AgentLoop<TResult> - holds all mutable state for a single run and drives the ToolLoopAgent. Configured via AgentConfig (name, model, systemPrompt, tools, reportTool, optional maxSteps, optional compactor). The loop stops when the report tool produces a result or maxSteps is hit. Overridable hooks: prepareStep (inject per-step messages/settings), onStepFinish (per-step side effects; default logs the step), snapshotPartial (expose a partial result on failure). The system prompt is fixed at construction - anything per-run belongs in the user prompt.
- Terminal errors:
NoAgentResultError, MaxStepsReached. A duplicate result call throws MultipleResultCalls, which is fixable: it is returned to the model and the first result is kept.
run / runLoop return AgentRunResult<TResult>: { result, conversation }, where conversation is the raw, uncompacted message stream.
-
AgentTool<TInput, TOutput, TLoop> - a thin wrapper over the AI SDK Tool with typed error handling and access to the loop via execute(input, loop). Failures are classified:
- Throw
FixableToolError (with a suggestFix() message) to feed an error back to the model and continue.
- Throw
FatalToolError to stop the loop and propagate.
- The
errorHandling policy (continue_unless_fatal default, or ) decides how exceptions are treated.
Model Registry (registry/)
ModelRegistry<TModel> manages all LLM instances with middleware for usage tracking, cost calculation, and monitoring. Models are defined as entries in MODEL_ENTRIES (primary providers) and OPENROUTER_MODEL_ENTRIES (OpenRouter fallback) in registry/model-entries.ts.
- Default model for tests and scripts: always
GEMINI_3_FLASH_PREVIEW (gemini-3-flash-preview). Never use older Gemini versions. Do not hardcode the full model list here - read model-entries.ts, since it changes often.
- Providers:
groqProvider, googleProvider, openRouterProvider - all lazy singletons (registry/providers.ts).
- Reasoning effort:
"none" | "low" | "medium" | "high", mapped to provider-specific params (Groq reasoningEffort, Google thinkingConfig.thinkingLevel).
- Cost tracking:
CostCollector hooks into the registry's monitoring callbacks to capture per-call token usage and cost (microdollars). Attach at construction time or pass to getModel(options, costCollector) for per-run attribution on a shared registry.
ObjectGenerator (object/)
The core structured-output engine that most other primitives build on. Takes a Zod schema and returns validated JSON from an LLM. Features:
- Zod schema validation of the output.
- Multimodal input: text + images + video (
VideoProcessor uploads to the Google GenAI Files API; only Google models pass the modelSupportsVideo check).
- Automatic retry with exponential backoff.
- Strips null bytes from responses (PostgreSQL compatibility).
- Tool support for agentic generation workflows.
Text Primitives (@autonoma/ai, text/)
| Class | Purpose |
|---|
AssertionSplitter | Split a compound assertion into atomic assertions (text/assertion-splitter.ts) |
Visual Primitives (@autonoma/visual-ai, visual/)
These live in @autonoma/visual-ai (they operate on Screenshot and pull in sharp). Import them from @autonoma/visual-ai, not @autonoma/ai.
| Class | Purpose |
|---|
VisualConditionChecker | Check whether a condition is met on a screenshot. Returns { metCondition, reason } |
AssertChecker | VisualConditionChecker with an assertion-specific prompt. Used by the assert command |
TextExtractor | Extract exact text values from a screenshot. Used by the read command |
VisualChooser | Given UI elements (with bounding boxes) + an instruction, pick the matching element |
Element Detection (@autonoma/visual-ai, freestyle/)
Also in @autonoma/visual-ai. PointDetector (abstract) locates a single pixel coordinate from a description; ObjectDetector (abstract) detects bounding boxes. They are used by the engine's click and type commands. Concrete implementations (e.g. GeminiComputerUsePointDetector, ObjectPointDetector, GeminiObjectDetector) live under freestyle/; resolution normalization is handled in the base classes. See the README if you need a specific detector.
Other Subsystems
compaction/ (@autonoma/agent-core, re-exported by @autonoma/ai) - message-compaction strategies consumed by the agent loop's compactor.
Evaluation Framework
The generic harness - Evaluation<TTestCase> (Vitest integration) and ModelEvaluation (per-model token/cost tracking) - is exported from @autonoma/ai/evaluation. The concrete visual eval suites (assertion accuracy, click/point detection accuracy, wait-condition accuracy) and the ScreenshotTestCaseLoader live in @autonoma/visual-ai's evals/, since they exercise the visual primitives.
For package exports, the live model list, and detailed usage examples, see packages/ai/README.md and packages/visual-ai/README.md.
