execution-agent

name	execution-agent
description	Always read this skill before editing any file under packages/engine/. Covers the platform-agnostic execution agent, command system, driver interfaces, and runner that engine-web (Playwright) and engine-mobile (Appium) extend.

Execution Agent (@autonoma/engine)

The execution agent is the core of test execution: a generic, platform-agnostic AI agent that the web and mobile engines extend. Everything is parameterized with TSpec (command spec) and TContext (driver context), so the same agent core works for both Playwright (web) and Appium (mobile).

It lives in the @autonoma/engine package. This skill is the conceptual map; the exhaustive file tree, package exports, and the full command table are maintained in packages/engine/README.md - treat that README as the source of truth and update it when exports or behavior change.

How the Agent Works

ExecutionAgent wraps the Vercel AI SDK's ToolLoopAgent. Each iteration:

1. Screenshot - capture the current screen state.
2. Inject context - package screenshot + instruction + steps-so-far into a user message.
3. LLM decides - the model picks a tool/command to call (or finishes).
4. Command executes - the chosen command runs against platform drivers.
5. Record step - save the attempt with before/after metadata.
6. Loop or stop - continue until execution-finished is called or maxSteps (default 50) is reached.

Loop detection: when the model finishes with success: false and its reasoning mentions "loop", "stuck", "no progress", "repeating", or "repeated", the run is flagged as a loop.

Where Things Live

packages/engine/src/
├── execution-agent/    # agent loop, LLM tools, memory, runner, generation-api
│   ├── agent/          # execution-agent.ts, execution-agent-factory.ts, execution-result.ts, tools/, memory/
│   ├── runner/         # execution-agent-runner.ts, artifacts, events
│   ├── generation-api/ # API-driven runner + persistence
│   └── local-dev/      # local-runner.ts, test-case loading, cost summary
├── commands/           # command base class + per-command directories
└── platform/           # driver interfaces + context (installer, image stream, video recorder)

Web and mobile packages (packages/engine-web, packages/engine-mobile) only implement the platform interfaces; all agent logic stays here.

Command Model

A command is described by a CommandSpec - three grouped types:

interface CommandSpec {
  interaction: string;   // command name (becomes the LLM tool name)
  params: object;        // what gets stored to execute the step
  output: { outcome: string };  // what the command returns
}

The abstract base class is Command<TSpec, TContext> (packages/engine/src/commands/command.ts), with:

• interaction - the command name (readonly literal).
• paramsSchema - a Zod schema for params.
• execute(params, context) - performs the action against the drivers in context.

Each command is split across files in its own directory under commands/commands/<name>/:

• <name>.def.ts - the CommandSpec interface + base param Zod schema. Pure types/schema, no runtime deps. Re-exported from command-defs.ts (the lightweight @autonoma/engine/command-defs entry).
• <name>.command.ts - the abstract Command subclass with execute(). Platform variants live alongside (e.g. web-click.command.ts, mobile-click.command.ts).
• <name>.tool.ts (under execution-agent/agent/tools/commands/) - a CommandTool subclass that exposes the command to the LLM via description(), inputSchema(), and extractParams().

Every command is exposed to the LLM, so the directories under commands/commands/ and the tools under execution-agent/agent/tools/commands/ are one-to-one: assert, click, drag, hover, navigate, read, refresh, save-clipboard, scroll, type.

LLM Tools (non-command)

Tool	Purpose
wait	Sleeps for N seconds. Not recorded as a step
ask-user	Asks a human for clarification via WebSocket; pauses until answered (only registered when a handler is provided)
execution-finished	Ends the test. Takes { success, reasoning }

Driver Interfaces (platform/drivers/)

Platform apps implement these; the agent only depends on the interfaces:

• ScreenDriver - screenshot, resolution.
• MouseDriver - click, scroll, hover, drag.
• KeyboardDriver - type, press, and related editing.
• ClipboardDriver - read the clipboard (backs save-clipboard).
• ApplicationDriver - wait until the app is stable.
• NavigationDriver - navigate, current URL, refresh.

See the README for the exact method signatures.

Memory and Variables

execution-agent/agent/memory/ holds a MemoryStore - a key/value store of values extracted during a run (e.g. by the read command). Commands reference stored values with {{variableName}} templates, resolved before execution. A test case can seed memory via credentials and recipeVariables.

Result Types

• StepAttempt<TSpec> - a single command attempt, discriminated on status:
  • GeneratedStep<TSpec> (status: "success") - execution output plus before/after metadata.
  • FailedStep<TSpec> (status: "failed") - the attempted interaction, the raw model input, the extracted params (absent if extraction itself threw), error/errorName, and the before metadata. Failed attempts are recorded, not discarded - they make the agent's botched attempts visible to users and to review.
• ExecutionResult<TSpec> - full result: generatedSteps (the successful subset), memory, success, finishReason: "success" | "max_steps" | "error", reasoning, final screenshot, conversation.
• LeanExecutionResult<TSpec> - network-safe version with large image buffers stripped.

Test Cases as Markdown

Test files use gray-matter frontmatter for parameters, with the body as the natural language prompt:

---
url: https://example.com
---
Navigate to the login page, enter "[e-mail]" and "password123", click Sign In, and assert the dashboard is visible.

Adding a New Command

Read an existing command (e.g. commands/commands/click/) before starting, then mirror the triplet:

1. <name>.def.ts - declare the CommandSpec interface (interaction, params, output) and a base param Zod schema. Add export * from "./commands/<name>/<name>.def" to commands/command-defs.ts.
2. <name>.command.ts - extend Command<YourSpec, YourContext>: set interaction, implement paramsSchema, and implement execute(params, context). Add a platform variant if behavior differs across web/mobile. Export it from commands/commands/index.ts.
3. <name>.tool.ts - extend CommandTool under execution-agent/agent/tools/commands/: implement description() (shown to the LLM), inputSchema(), and extractParams(). Export it from that directory's index.ts and wire it into the agent factory so the LLM can call it.
4. UI rendering (@autonoma/blacklight) - add a CommandUI for the new interaction so recorded steps render with a label, icon, and badge in the app. Edit packages/blacklight/src/components/commands/display-command.tsx: import a Phosphor icon, define a <name>UI: CommandUI object (name, an instruction(params) that renders the human-readable step from the stored params, iconComponent, a color from command-colors.ts, and badgeClassName), and add it to the displayCommand map. This is not compiler-enforced - an unmapped interaction silently falls back to the "Unknown" command UI (stepInstruction/StepIcon resolve via getUI, which defaults), so it is easy to forget. The map is consumed by apps/ui step views (e.g. step-card.tsx, reproduction-steps.tsx).

Extending for a New Platform

1. Implement all driver interfaces (ScreenDriver, MouseDriver, etc.) with your platform's SDK.
2. Create an Installer subclass that builds the context (drivers + image stream + video recorder).
3. Create an ExecutionAgentFactory subclass that calls prepareContext() for platform-specific setup.
4. Create a runner entry point that wires everything together.

---

For the full file tree, package exports, and the complete command table, see packages/engine/README.md.