edlsh/omp-computer-use

edlsh profile

Latest recorded source activity: July 13, 2026 at 09:17

SkillsMP catalog refreshed: October 26, 2026 at 11:37

skills collected: 1

GitHub stars: 0
GitHub forks: 1

Repository README

Repository README source: README.md

Read the source README

Expand full README

# omp-computer-use Native [Oh My Pi](https://github.com/can1357/oh-my-pi) extension for state-scoped desktop control on macOS and Windows. `omp-computer-use` lets OMP agents inspect application windows, find controls, click, type, scroll, wait for UI changes, and verify observable outcomes. It keeps complete immutable observations behind compact `stateId`-scoped views so concurrent calls cannot silently act on stale UI. The macOS helper requires macOS 14 or newer. ## When to use it Prefer a reliable API, MCP server, file operation, terminal command, or OMP's dedicated browser tool when one fits. Computer use is for software whose usable interface is the app on screen. The extension adds: - first-class OMP tools with read/write/exec approval tiers; - macOS Accessibility and ScreenCaptureKit integration; - Windows UI Automation and native input integration; - immutable UI states with stale-write rejection; - cached search, expansion, inspection, and paginated text; - checked action batches, semantic postconditions, and successor diffs; - optional OCR-backed fused observations; - managed CDP browser roots sharing the same state model. ## Install ```bash omp install git:github.com/edlsh/omp-computer-use ``` Start OMP interactively after installation: ```bash omp ``` On macOS, the first computer-use tool call registers the native helper and asks for: - **Accessibility** - **Screen Recording**, called **Screen & System Audio Recording** on newer macOS releases Grant both permissions to: ```text /Applications/pi-computer-use.app ``` The helper retains the upstream app identity so upgrades do not unnecessarily replace the macOS permission target. Enable each toggle, return to OMP, and choose **Recheck (restarts helper)**. On Windows, run OMP in an unlocked interactive desktop session. The package does not include a Windows executable; its first live tool call builds the included Rust helper, so Rust and `cargo` must be on `PATH`. The helper is then installed under: ```text %USERPROFILE%\.omp\agent\helpers\omp-computer-use\windows-bridge.exe ``` Use `/computer-use` inside OMP to show the effective configuration and its sources. ## Core workflow ```text find_roots → observe_ui → search_ui / expand_ui / inspect_ui → act_ui ``` An `@eN` element ref belongs only to the `stateId` that returned it. Continue from the successor state after every mutation; re-observe after uncertain outcomes or external UI changes. Main tools: - `find_roots` - `observe_ui` - `search_ui` - `expand_ui` - `inspect_ui` - `act_ui` - `read_text` - `wait_for` - `launch_browser` - `navigate_browser` - `evaluate_browser` The included `omp-computer-use` skill teaches OMP the state/ref workflow automatically. ## Configuration User configuration: ```text ~/.omp/agent/extensions/omp-computer-use.json ``` Project configuration: ```text <project>/.omp/computer-use.json ``` Example: ```json { "browser_use": true, "headless": false, "cursor_overlay": true } ``` Project settings override user settings. `PI_COMPUTER_USE_BROWSER_USE`, `PI_COMPUTER_USE_HEADLESS`, and `PI_COMPUTER_USE_CURSOR_OVERLAY` override both. See [Configuration](./docs/configuration.md) for details. ## Development ```bash bun install bun run test ``` Additional platform checks: ```bash bun run test:platform bun run test:windows-scripts ``` See: - [Usage](./docs/usage.md) - [Architecture](./docs/architecture.md) - [Configuration](./docs/configuration.md) - [Development](./docs/development.md) - [Troubleshooting](./docs/troubleshooting.md) - [Contributing](./CONTRIBUTING.md) ## Upstream This project is an OMP-native port of [`injaneity/pi-computer-use`](https://github.com/injaneity/pi-computer-use). It preserves the upstream native helpers, state model, and MIT license while replacing the host integration, package metadata, paths, approval policy, runtime guidance, and Bun lifecycle for Oh My Pi. ## License MIT

Loading the repository README…

Skills in this repository

Software Developers

1 occupation category · 100% classified

skill

occupation

description

updated

omp-computer-use

Software Developers

Use OMP's state-scoped desktop and managed-browser tools to observe and control macOS or Windows apps.

2026-07-13

Showing 1 of 1 collected skill.

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/edlsh/omp-computer-use.