kova-operator

# Kova Operator ## Mission Use Kova to test OpenClaw as users actually run it. Kova is not a unit-test runner and not an OCM test suite. OCM is the lab control plane; OpenClaw is the product under test. Kova should produce evidence a maintainer can act on: - startup time, listening time, health readiness - gateway, CLI, package-manager, runtime-staging, plugin, browser, and provider CPU/RSS attribution - plugin load and missing dependency errors - dashboard/TUI/API responsiveness - agent turn breakdown: pre-provider OpenClaw time, provider time, post-provider time, cold/warm delta - provider failure containment: timeout, malformed response, streaming stall, recovery, leaked children, gateway health - repeat-run stats, baseline comparisons, and release gate verdicts Do not overstate causality. If OpenClaw did not emit enough diagnostic spans, say Kova proved the outside-in timing but internal attribution is missing. ## First Steps 1. Find Kova: ```sh command -v kova || true test -f bin/kova.mjs && node bin/kova.mjs version ``` Use `kova` when installed. Inside a Kova checkout, use: ```sh node bin/kova.mjs <command> ``` 2. Ensure OCM is available before real execution: ```sh command -v ocm ocm env list ocm runtime list ``` If the `ocm-operator` skill is available, use it before running Kova scenarios that build runtimes, clone existing user envs, inspect services/logs, or clean up envs. 3. Set up Kova: ```sh kova setup kova self-check ``` For scripts/CI: ```sh kova setup --ci --json kova self-check --json ``` If using the repo checkout, replace `kova` with `node bin/kova.mjs`. ## Install OCM Precisely Kova real execution requires OCM. If OCM is missing, install it before running benchmarks: ```sh command -v ocm || curl -fsSL https://raw.githubusercontent.com/shakkernerd/ocm/main/install.sh | bash export PATH="$HOME/.local/bin:$PATH" ocm --version ocm env list ocm runtime list ``` If installing Kova from its installer, OCM can be installed or required there: ```sh curl -fsSL https://raw.githubusercontent.com/shakkernerd/Kova/main/install.sh | bash -s -- --install-ocm --require-ocm ``` For Codex agents working from this repo, use the repo-local OCM operating skill at `.agents/skills/ocm-operator` when direct OCM operations are needed. ## Safety Rules - Prefer mock auth first. It isolates OpenClaw latency from provider latency. - Run live auth only as a separate lane and mark it environment-dependent. - Prefer `local-build:<repo>` for release-like validation of local OpenClaw source. - Do not use source/dev commands as proof a published package will work. - Do not mutate durable user envs directly. Use Kova clone/source flows. - Use `--execute` only when a real run is intended. - Keep failing envs only when inspection is needed; otherwise let Kova clean up. - Report OpenClaw failures as OpenClaw failures, not OCM failures, unless lab provisioning itself blocked the run. ## OpenClaw Repo Handling When the user asks to benchmark OpenClaw, first decide which OpenClaw source is under test: 1. Use the explicit repo path if the user gives one. 2. If the current directory is an OpenClaw checkout, use it. 3. If a nearby checkout exists, such as `../openclaw`, use it only after confirming the path and commit. 4. If no checkout exists, clone a disposable copy into `/tmp`, not inside Kova: ```sh repo="/tmp/kova-openclaw-$(date +%Y%m%d%H%M%S)" git clone https://github.com/openclaw/openclaw.git "$repo" git -C "$repo" checkout main git -C "$repo" pull --ff-only ``` Before benchmarking a checkout, record: ```sh git -C "$repo" status --short git -C "$repo" rev-parse --abbrev-ref HEAD git -C "$repo" rev-parse --short HEAD git -C "$repo" log -1 --format='%h %ci %s' ``` Do not mutate a user working tree unless the user asked for that. For benchmarking moving targets, prefer disposable clones or worktrees under `/tmp/kova-openclaw-*`. Use release-shaped Kova targets for local source: ```sh kova matrix run --profile diagnostic --target local-build:"$repo" --execute --json ``` Do not use OpenClaw source/dev commands as proof that a package or release will work. Source commands can hide missing package files, prepack failures, bundled extension issues, or runtime dependency layout problems. ## Build Blocker Triage A failed `local-build:<repo>` run is not automatically a finished benchmark. The agent must determine whether the failure is: - an OpenClaw packaging/prepack bug - a missing dependency/install problem - stale local artifacts - a dirty checkout problem - a platform/toolchain problem - a Kova/OCM harness problem If Kova reports a build/prepack blocker, inspect and reproduce the failing step inside the OpenClaw repo before returning: ```sh git -C "$repo" status --short node --version pnpm --version || corepack enable pnpm -C "$repo" install --frozen-lockfile pnpm -C "$repo" build pnpm -C "$repo" pack --pack-destination /tmp/kova-openclaw-pack-check ``` If the Kova/OCM error names a specific script, run that script directly too: ```sh pnpm -C "$repo" <script-name> ``` Examples: ```sh pnpm -C "$repo" canvas:a2ui:bundle pnpm -C "$repo" runtime:postbuild pnpm -C "$repo" test-built-bundled-channel-entry-smoke ``` Use the direct script result to classify the issue: - Direct script fails too: OpenClaw script/build bug. Report exact command and error. - Direct script passes but `pnpm pack` or OCM `runtime build-local` fails: OpenClaw packaging/prepack path bug. Report that difference. - `pnpm install` fails: dependency/toolchain problem. Report lockfile, Node, pnpm, and first failing package. - Kova cannot create/remove envs or runtimes after OpenClaw packs correctly: likely harness/OCM problem. Report as BLOCKED with OCM evidence. Do not stop at "build failed" when a simple build step fails. Try to isolate the smallest failing OpenClaw command and include the exact failure line. A blocked benchmark is acceptable only after the blocker is classified. If stale generated artifacts are plausible, clean only disposable clones: ```sh git -C "$repo" clean -fdX pnpm -C "$repo" install --frozen-lockfile ``` Never run `git clean`, `git reset`, or destructive cleanup in a user's dirty working checkout unless explicitly approved. ## How To Choose Scenarios From User Intent When the user describes "what kind of thing to test", map the intent to Kova scenarios or tags. Prefer the narrowest command that answers the request. Use `local-build:<repo>` when testing a local OpenClaw checkout and `runtime:stable`, `npm:<version>`, or `channel:<name>` when testing installed/published behavior. Before running unfamiliar requests, inspect the registry: ```sh kova plan --json kova matrix plan --profile smoke --target runtime:stable --json ``` Intent map: ```text release readiness / can we ship profile: release command: kova matrix run --profile release --target local-build:<repo> --execute --gate --json local source as real release package profile: diagnostic command: kova matrix run --profile diagnostic --target local-build:<repo> --execute --json quick confidence / smoke profile: smoke command: kova matrix run --profile smoke --target <target> --execute --json startup / gateway readiness / memory / CPU scenarios: release-runtime-startup, gateway-performance command: kova run --target <target> --scenario gateway-performance --execute --json deep CPU/heap/function profiling scenarios: gateway-performance, agent-cold-warm-message, dashboard-session-send-turn command: kova run --target <target> --scenario gateway-performance --execute --deep-profile --json slow first reply / agent latency scenarios: dashboard-session-send-turn, agent-cold-warm-message, agent-gateway-rpc-turn command: run dashboard-session-send-turn first, then agent-cold-warm-message dashboard messaging scenarios: dashboard-readiness, dashboard-session-send-turn command: kova run --target <target> --scenario dashboard-session-send-turn --execute --json TUI input / terminal message path scenarios: tui-responsiveness, tui-message-turn command: kova run --target <target> --scenario tui-message-turn --execute --json OpenAI-compatible API path scenario: openai-compatible-turn command: kova run --target <target> --scenario openai-compatible-turn --execute --json plugins overall tag: plugins command: kova matrix run --profile release --target <target> --include tag:plugins --execute --json bundled plugin startup / missing runtime deps scenarios: bundled-plugin-startup, plugin-missing-runtime-deps command: kova run --target <target> --scenario bundled-plugin-startup --execute --json legacy bundled runtime dependency staging scenario: bundled-runtime-deps command: kova run --target <historical-target> --scenario bundled-runtime-deps --execute --json external plugin lifecycle scenarios: plugin-external-install, plugin-lifecycle, plugin-update, plugin-remove command: kova matrix run --profile release --target <target> --include scenario:plugin-lifecycle --execute --json bad plugin manifest scenario: plugin-bad-manifest command: kova run --target <target> --scenario plugin-bad-manifest --execute --json provider timeout / malformed / streaming stall / recovery tag: provider-failure command: kova matrix run --profile release --target <target> --include tag:provider-failure --execute --json model/provider listing or auth states scenarios: provider-models, agent-auth-missing command: kova run --target <target> --scenario provider-models --execute --json existing user upgrade scenarios: upgrade-existing-user, upgrade-from-2026-4-20, upgrade-from-2026-4-24 command: kova run --scenario upgrade-existing-user --source-env <env> --from npm:<old> --target npm:<new> --execute --json upgrade stable channel to beta/local build scenarios: upgrade-stable-channel-to-beta, upgrade-stable-channel-to-local-build command: kova matrix run --profile channel-upgrade --from channel:stable --target channel:beta --execute --json workspace/filesystem pressure scenarios: workspace-scan-pressure, gateway-performance with slow-filesystem state command: kova run --target <target> --scenario workspace-scan-pressure --execute --json long-running memory/session pressure profile: soak scenarios: soak, agent-long-session command: kova matrix run --profile soak --target <target> --execute --json browser automation scenario: browser-automation-smoke command: kova run --target <target> --scenario browser-automation-smoke --execute --json MCP runtime scenario: mcp-runtime-start-stop command: kova run --target <target> --scenario mcp-runtime-start-stop --execute --json media understanding timeout scenario: media-understanding-timeout command: kova run --target <target> --scenario media-understanding-timeout --execute --json network offline containment scenario: agent-network-offline command: kova run --target <target> --scenario agent-network-offline --execute --json ``` If the user gives several intents, prefer a matrix with `--include` filters when possible. If no existing scenario matches, say the current Kova registry does not cover that intent and propose the closest scenario or a new scenario name. ## Core Workflows ### Benchmark a local OpenClaw checkout like a release Use this first when validating a branch or commit: ```sh kova matrix run \ --profile diagnostic \ --target local-build:/path/to/openclaw \ --execute \ --json ``` This covers release-shaped packaging, startup, runtime deps, bundled plugins, gateway readiness, dashboard/API/agent surfaces, timelines when available, and resource attribution. ### Fast smoke against an existing runtime ```sh kova matrix run \ --profile smoke \ --target runtime:stable \ --execute \ --json ``` Use this for quick confidence when an OCM runtime already exists. ### Release gate ```sh kova matrix run \ --profile release \ --target local-build:/path/to/openclaw \ --execute \ --gate \ --json ``` Gate verdicts are `SHIP`, `DO_NOT_SHIP`, `PARTIAL`, or `BLOCKED`. A partial filtered run can reject a release but cannot approve the whole release. ### Investigate slow OpenClaw replies Run both dashboard and CLI agent paths: ```sh kova run \ --target local-build:/path/to/openclaw \ --scenario dashboard-session-send-turn \ --execute \ --json kova run \ --target local-build:/path/to/openclaw \ --scenario agent-cold-warm-message \ --execute \ --json ``` Look for: - total agent turn time - pre-provider OpenClaw time - provider duration / first byte / final response - post-provider cleanup - cold vs warm delta - gateway health during and after the turn - leaked child processes - OpenClaw timeline spans if available If provider time is tiny and pre-provider time dominates, report that OpenClaw delayed before provider work. ### Provider failure containment ```sh kova matrix run \ --profile release \ --target runtime:stable \ --include tag:provider-failure \ --execute \ --json ``` This exercises slow provider, timeout, malformed response, streaming stall, concurrent pressure, and recovery scenarios. Verify the gateway remains healthy and child processes do not leak. ### Existing-user upgrade ```sh kova run \ --scenario upgrade-existing-user \ --source-env <existing-env-name> \ --from npm:<old-version> \ --target npm:<new-version> \ --execute \ --json ``` Use durable envs only as clone sources. Never mutate the source env directly. ### Repeat benchmarks and baselines ```sh kova matrix run \ --profile smoke \ --target local-build:/path/to/openclaw \ --repeat 3 \ --execute \ --json ``` Only save a baseline after a human reviews a passing, stable run: ```sh kova matrix run \ --profile smoke \ --target local-build:/path/to/openclaw \ --repeat 3 \ --execute \ --save-baseline \ --reviewed-good \ --json ``` Use baseline comparisons to catch startup, RSS, CPU, event-loop, runtime-deps, and agent-latency regressions. ### Benchmark a time window or commit range When the user asks to benchmark "between", "since", "before/after", "last N commits", "from date A to date B", or "over a period", produce a comparison. Do not only report separate runs. Required flow: 1. Identify the baseline commit and latest/end commit. 2. Run the same Kova scenario/profile on both commits. 3. Use the same machine, auth mode, profile, scenario/state, target style, and `--repeat` count. 4. Compare the generated JSON reports. 5. Report deltas and verdict. Use disposable OpenClaw worktrees or clones: ```sh base_repo="/tmp/kova-openclaw-base-<shortsha>" head_repo="/tmp/kova-openclaw-head-<shortsha>" git clone <openclaw-remote-or-source> "$base_repo" git clone <openclaw-remote-or-source> "$head_repo" git -C "$base_repo" checkout <baseline-sha>