computer-use

name	computer-use
description	Control local desktop applications through Computer Use for tasks that require reading or operating app UI. Prefer purpose-built connectors, APIs, or CLIs when available.

node_repl + @qwen-code/cua-sdk (Computer Use)

• Use node_repl (JavaScript) for all Computer Use actions.
• Do not use other technologies besides node_repl for computer interactions unless specifically requested by the user. This includes AppleScript, osascript, JXA, System Events, and synthesized input.
• Prefer a dedicated plugin or skill when it can complete the task; use Computer Use for app interactions that are not exposed through a more specific interface.
• Use only the typed ComputerUse API. Do not use a generic SDK callTool, direct CuaDriver imports, or a Qwen global bridge.
• node_repl state persists across calls.
• Use nodeRepl.write(...) for text output. It takes a string, so wrap objects with JSON.stringify(...).

Install automatically

Run qwen mcp list to check whether the node-repl server is configured. If it is not configured, run both commands yourself:

qwen mcp add --scope user node-repl npx -y @qwen-code/node-repl-mcp@0.1.1
npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.2

Tell the user to restart Qwen Code after adding the MCP server, then stop. Do not ask the user to copy or run the commands.

If node_repl is available but the SDK import fails, run the SDK installation command yourself from the current workspace, then retry the import:

npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.2

Bootstrap

Import the SDK once per fresh node_repl kernel:

globalThis.computer = await (
  await import('@qwen-code/cua-sdk/computer-use')
).ComputerUse.create();

API surface

type WindowTarget = { pid: number; windowId: number };
type ElementTarget = {
  pid: number;
  windowId?: number;
  elementToken: string;
};
type CoordinateTarget = WindowTarget & { x: number; y: number };
type PointOrElementTarget = CoordinateTarget | ElementTarget;
type App = {
  name?: string;
  bundle_id?: string;
  pid?: number;
  running?: boolean;
  launch_path?: string;
};
type Window = {
  window_id: number;
  title?: string;
  is_on_screen?: boolean;
  on_current_space?: boolean;
};
type Element = {
  element_token?: string;
  role?: string;
  label?: ;
  ?: ;
  ?: [];
};

  = {
  :  <[]>;
  :  <[]>;
  :  <{
    : ;
    : [];
    ?: ;
    ?: { : [] };
  }>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
  :  <>;
};

Workflow

1. Initialize

Resolve the exact running application and window named by the task. Filter inside node_repl; do not print the entire application list:

var apps = await computer.listApps();
var matches = apps.filter(
  (app) => app.name === 'Target App' || app.bundle_id === 'com.example.target',
);
nodeRepl.write(JSON.stringify(matches));

After choosing the application from returned metadata, list only its windows:

var pid = matches[0].pid;
var windows = await computer.listWindows({ pid });
nodeRepl.write(JSON.stringify(windows));

Choose the window from returned metadata, then get its current accessibility state:

var target = { pid, windowId: windows[0].window_id };
var state = await computer.observeWindow({ ...target, forceFull: true });
nodeRepl.write(state.text);

Never guess a PID, window ID, coordinate, or element token. ComputerUse discovers running applications but does not launch them; if necessary, start the application from node_repl with ordinary Node.js process APIs, then refresh the application and window lists.

2. Act and get the latest state

Choose only the action needed for the user's task. Prefer current element_token values over coordinates. Pass an observed element_token as the camel-case elementToken action field:

await computer.setValue({
  ...target,
  elementToken,
  value: 'hello',
});

state = await computer.observeWindow({
  ...target,
  baseRevisionId: state.revisionId,
});
nodeRepl.write(state.text);

After one or more actions, always observe the exact window before deciding what to do next. If the updated state shows the requested result, stop acting, clean up, and answer the user. If the UI does not behave as expected, get a fresh full state before choosing a different action.

Use a secondary action only when the current accessibility state advertises that exact action. Prefer accessibility text for efficiency; use a screenshot when it is incomplete or visual layout matters.

Reading screenshots

var state = await computer.observeWindow({
  ...target,
  forceFull: true,
  includeScreenshot: true,
});

for (var image of state.screenshot?.images ?? []) {
  if (image?.dataBase64 && image?.mimeType) {
    await nodeRepl.emitImage(
      `data:${image.mimeType};base64,${image.dataBase64}`,
    );
  }
}

Finish

When the task is complete, close the SDK client:

await computer.close();
globalThis.computer = undefined;

Call node_repl_reset when no other REPL state is needed.