koto

name	koto
description	Drive a Linux/Hyprland desktop keyboard-first through the koto CLI, which executes basm (beckett assembly). Use for any GUI computer-use task on this machine — launching and arranging windows, typing into applications, operating web pages through a real browser, or reading structured observations instead of screenshots.

Using koto

You drive a Linux/Hyprland desktop by invoking the koto CLI. koto executes basm (beckett assembly): explicit keyboard chords, strict window selectors, bounded control flow, and one final observation instead of a screenshot stream.

Invocation

koto <instruction>...                  # inline mode, argv token stream
koto --script FILE                     # script mode, one instruction per line
koto --script FILE -- ARG1 ARG2        # args bind to %1, %2

Useful globals: --allow a,b / --deny a,b (capabilities), --timeout 10s, --budget-ops 256, --budget-time 120s, --format agent|json|raw|quiet, --observe auto|text|image|both, --dry-run, --explain.

Work inline, one step at a time. Run a few instructions, end with end image (or end text), look at what came back, then decide the next call. Scripts are for flows you have already worked out and intend to repeat — not for finding your way around a page you have never seen. A script that guesses at six steps fails on step two and tells you nothing about why.

end observes and stops the program; it does not tear down the world. The browser survives between invocations (see below), so the next call picks up exactly where this one stopped.

Instruction set

Family	Mnemonics
Input	key <chord>, tap <key>, hold <mod>..., release <mod>..., type "<text>", paste "<text>", click <sel> (cap: pointer), scroll <dir> [n]
Observe	end [text|image|json|both|silent], see [$reg] [text|image], peek <field>, ocr (needs a prior see image / end image in the same run — it re-reads the last capture and fails with exit 7 if there is none)
Wait	wait <dur>, wait window <sel>, wait gone <sel>, wait title <pat>, wait text <pat>, wait idle [<dur>], wait exit <pid>; all accept timeout <dur>
Window	focus <sel>, ws <n|+1|-1|prev>, send ws <n>, close [sel], float, tile, full, pin, swap <l|r|u|d>, move <l|r|u|d>, monitor <name>, list <windows|workspaces|monitors|clients>
Process	spawn <cmd>... (cap: spawn), kill <sel|scope>
Terminal	pane new [name], pane send "<text>", pane run "<cmd>", pane read [n], pane wait <pat>, pane kill [name]
Web	web attach [bw|<browser>|<sel>] (cap: web), web goto <url>, web back/forward/reload, web click <ref|css|text=> [newtab], web type <target> "<text>" [append], web fill <target> "<text>", web scroll <n>, web hover <target>, web press <key>, , , , , , / / / , / / , , , (cap: web.eval), + (cap: web.login), (cap: web.download), , , , , . The session persists across invocations; ends it.

Capabilities you may need in require / --allow: input, window, spawn, exec, web, web.eval, web.login, web.download, pointer, fs. Note pane run requires exec — spawn alone is not enough — and web eval requires web.eval on top of web.

Web engines and the persistent session

web attach picks the engine, and both engines' browsers persist between koto invocations. Navigate in one call, click in the next, fill a form in a third: the page, the tabs, and the scroll position stay where you left them. Only the first attach pays for a browser launch (~5s); the rest connect in about a second. koto --web-status reports both sessions, koto --web-stop closes them.

Bare web attach (or web attach <browser>) drives Chrome over CDP in a koto-owned profile and supports only goto/read/click/fill/wait/eval. web attach bw uses BetterWright 1.6.3 (needs node and npm i -g betterwright@1.6.3; an older install is refused at attach) and exposes the whole engine, one koto action per betterwright function. Attach options: profile=<name> (separate cookie jar), session=<name> (parallel lane in one browser), platform=linux|macos|windows (identity fingerprint; koto defaults to linux so the headed browser renders at host scale instead of betterwright's 2x-Retina macOS default).

• Every action answers "where am I now". Success returns a page <id> <url> "<title>" line; type/fill also echo the field's actual value (passwords redact). A missed target returns the page it missed on, the nearest live candidates with refs, and a help: line — read the error before re-reading the page; it usually already contains your next target.
• Read before acting. web read is snapshot({interactive:true}) — a pruned accessibility tree with [ref=eN] markers that work directly as targets. full reads content wholesale, diff shows only what changed since the last same-shaped read, ref=eN/selector=<css> scope to a subtree, urls keeps link hrefs. Refs are reassigned on every read and go stale when the page changes, so re-read before acting. A scoped read needs to match exactly one element: too many is exit 6, none is exit 3, and both errors name the fix. Escalate read → read full → shot annotate (boxes every interactive element with its ref) only as far as the question needs.
• Visible actions are human-shaped. web click, web type, and web scroll run betterwright's human.* helpers (curved pointer, bounded key timing). web fill is the precise, instant locator.fill for when shaping does not matter. web press/web hover/web select cover keys, hovers, and dropdowns.
• Tabs are a session, and the session outlives the invocation. web open [url], web use <tab>, web close, web pages; web click <link> newtab Ctrl+clicks a link into a background tab and returns the new tab's summary. Tabs and the current page survive between koto runs until --web-stop, so a later invocation can web read or web click without re-navigating. Refs (eN) are the exception — they go stale on any page change, so re-read.
• Verify, don't infer. web overlays dismisses cookie/promo layers, web controls reports exact form-control state, reports what is actually playing. arms the next JS dialog before the click that triggers it.

The two engines keep separate profiles and separate logins.

Targets for web click, web type, web fill, web hover, web select, and web wait are, in order of preference: eN (a ref from the last bw web read), text=save draft (matches the visible words on a button, link, or label — survives a class rename), or a CSS selector. A target that matches nothing is exit 3, never a silent no-op.

web eval runs in betterwright's worker, not the page — there is no document. Its globals are page, pages, snapshot, human, credentials; reach the DOM through page.evaluate(() => document…).

Reach for web eval only when no instruction covers what you need. A flow written entirely in evals is a JavaScript program wearing koto as a hat: it gives up the observation ladder, the exit contract, and the capability gate, which are the reasons to use koto at all.

Selectors

Comma is AND. = is exact, ~ is a regex.

class=chromium      title~"^nvim"     addr=0x55f1a2    pid=4821
scope=app-foo.scope ws=3              role=button      name="Save"
web=#login          focused           last

Zero matches is exit 3; more than one is exit 6 (except close, kill, list, which accept sets). Ambiguity is an error, never a coin flip.

Lexical rules that bite

• Script mode: one instruction per line; ; starts a comment outside quotes.
• Inline mode is an argv token stream. A bare non-mnemonic token implies key, so koto super enter is one chord and koto enter enter is two presses.
• end collides with the End key. At a bare instruction boundary end halts and observes. After key or a modifier (ctrl end, key end) it is the End key.
• while must carry max <n>. There is no unbounded loop and no flag to enable one.
• type synthesises keysyms and is layout-aware; use paste above ~200 chars or for anything outside the active keymap.

Registers

$0–$9 general purpose, $? last status, $w focused window record (dotted: $w.class), $out last observation text, $n loop counter, %1–%9/%* script args, $env.NAME environment passthrough. Registers persist across --script files in one invocation.

Exit codes

0	success	1	assertion failed	2	timeout	3	selector matched nothing
4	budget exceeded	5	capability denied	6	selector ambiguous	7	observation unavailable
8	basm parse error	9	backend unavailable	64	internal error

Working style

• Keyboard first. click needs the pointer capability and always warns in the trace. Coordinates never appear in a well-written script.
• Screenshots are available. end image (or end both) captures the screen and returns it to you as an image. On the bw engine it captures the page itself, even occluded.
• Structured before pixels. The observation ladder runs CDP/BetterWright → AT-SPI → tmux → Hyprland metadata → screencopy → OCR. end text and web read cost far less than an image. Reach for end image when the rungs above fail.
• Settle explicitly. wait idle 200ms after a window-opening chord is exact (it reads Hyprland's event socket), unlike a sleep.
• list clients writes JSON to $out and is the cheapest way to build a world model.
• Start every script with require, so a policy violation fails before instruction zero rather than halfway through a mutation.

Worked example

require input,window,spawn
spawn alacritty --title KOTO_EVAL_WIN_1
wait window title~"KOTO_EVAL_WIN_1" timeout 10s
send ws 2
focus title~"KOTO_EVAL_WIN_1"
float
end text