openclaw-testing

# OpenClaw Testing Use this skill when deciding what to test, debugging failures, rerunning CI, or validating a change without wasting hours. ## Read First - `docs/reference/test.md` for local test commands. - `docs/ci.md` for CI scope, release checks, Docker chunks, and runner behavior. - Scoped `AGENTS.md` files before editing code under a subtree. ## Default Rule Prove the touched surface first. Do not reflexively run the whole suite. Route by source trust first, then required environment. Only trusted source may run locally; never execute untrusted repository tooling locally. Trusted development tests, changed gates, typecheck/lint, and builds run locally by default, including broad suites when they are the proportional proof. Use a remote backend only when the environment is part of the proof: clean-machine, install/package, Docker, E2E, live, desktop, or cross-platform work, or when the operator explicitly requests remote proof. Do not use Crabbox merely as generic compute offload. Untrusted contributor or fork code must use secretless fork CI or sanitized direct AWS Crabbox; never sync or run it on the credential-hydrated Blacksmith workflow. Do not pre-warm for anticipated work. Acquire the backend lazily when the first environment-sensitive command is ready to run, save its id, reuse it for later remote commands, and stop it before handoff. A single late remote command can remain a one-shot. For untrusted proof, switch to a clean trusted `main` checkout and lazily warm direct AWS with an installed trusted Crabbox binary. Do not execute the untrusted checkout's wrapper or config locally: ```bash cd <trusted-openclaw-main> env -u CRABBOX_AWS_INSTANCE_PROFILE \ crabbox config show --json | \ jq -e '.aws.instanceProfile == ""' >/dev/null env -u CRABBOX_AWS_INSTANCE_PROFILE \ -u CRABBOX_TAILSCALE \ -u CRABBOX_TAILSCALE_AUTH_KEY \ -u CRABBOX_TAILSCALE_AUTH_KEY_ENV \ -u CRABBOX_TAILSCALE_EXIT_NODE \ -u CRABBOX_TAILSCALE_EXIT_NODE_ALLOW_LAN_ACCESS \ -u CRABBOX_TAILSCALE_HOSTNAME_TEMPLATE \ -u CRABBOX_TAILSCALE_TAGS \ crabbox warmup \ --provider aws \ --network public \ --tailscale=false \ --tailscale-exit-node= \ --tailscale-exit-node-allow-lan-access=false \ --keep \ --timing-json crabbox inspect --provider aws --id <cbx_id> --json | \ jq -e '.network == "public" and .tailscale == null' >/dev/null ``` Bind the returned lease to one immutable reviewed head SHA; never repurpose a trusted or previously hydrated lease, and stop/rewarm if the head changes. Record the reviewed PR's full head SHA with `gh pr view <number> --repo <owner/repo> --json headRefOid --jq .headRefOid`. Every untrusted AWS run must override the repo env allowlist, skip Actions hydration, and upload the trusted bootstrap script from clean `main` alongside `--fresh-pr`. The script bypasses raw-box JavaScript preflight, proves the identity boundary, installs pinned Node/pnpm, verifies the exact SHA and package-manager pin, isolates `HOME`, installs dependencies, then runs the requested test command: ```bash env -u CRABBOX_AWS_INSTANCE_PROFILE \ CRABBOX_ENV_ALLOW=CI \ crabbox run \ --provider aws \ --id <cbx_id> \ --fresh-pr <owner/repo#number> \ --no-hydrate \ --timing-json \ --script scripts/crabbox-untrusted-bootstrap.sh -- \ <expected_head_sha> /usr/local/bin/pnpm test <path-or-filter> # After all proof: env -u CRABBOX_AWS_INSTANCE_PROFILE \ crabbox stop --provider aws <cbx_id> ``` Once remote proof starts, save the returned id, reuse it for later remote work, sync the current checkout on every run, and stop it before handoff. 1. Inspect the diff and classify the touched surface: - trusted development: run the smallest proportional local test, changed gate, typecheck/lint, or build; broaden locally when the contract requires it - clean-machine, install/package, Docker, E2E, live, desktop, or cross-OS proof: acquire the safe remote backend selected by source trust - `check:changed` classifies and runs the required local typecheck/lint/guard plan when dependencies are ready - direct AWS Crabbox proof: pass `--provider aws`; untrusted code also requires the sanitized invocation above - workflow-only: `git diff --check`, workflow syntax/lint (`actionlint` when available) - docs-only: `pnpm docs:list`, docs formatter/lint only if docs tooling changed or requested 2. Reproduce narrowly before fixing. 3. Fix root cause. 4. Rerun the same narrow proof. 5. Broaden only when the touched contract demands it. ## Guardrails - Do not kill unrelated processes or tests. If something is running elsewhere, treat it as owned by the user or another agent. - Run trusted development tests, checks, and builds locally with scope proportional to the touched contract. Untrusted repository tooling never runs locally. Remote proof requires a remote-environment or isolation reason. - Prefer GitHub Actions for release/Docker proof when the workflow already has the prepared image and secrets. - Standing up a local container Gateway for UI proof goes through `scripts/docker/setup.sh` (see `docs/install/docker.md`). It seeds `gateway.controlUi.allowedOrigins` for the published host port; a hand-rolled `docker run` skips that and the dashboard dead-ends on "Browser origin not allowed" with the Gateway logging `code=4008 reason=connect failed`. Never reuse the Compose defaults for a proof: they bind-mount the operator's real `~/.openclaw` and claim port 18789. - Use standard Git commands when committing; stage only your files. - If dependencies are missing on the selected host, run `pnpm install`, retry once, then report the first actionable error. - Codex and other linked/sparse worktrees may run local `pnpm test*` and `pnpm check*` when the dependency install is ready. If pnpm would reconcile a shared install, use `node scripts/run-vitest.mjs` or `node scripts/check-changed.mjs` to bypass that package-manager preflight. For actual remote proof, invoke `node scripts/crabbox-wrapper.mjs` directly rather than local `pnpm crabbox:run`. - For remote proof, use the Crabbox wrapper first, but name the actual backend. Direct AWS Crabbox uses `provider=aws` and `cbx_...` ids. Delegated Blacksmith Testbox through Crabbox uses `provider=blacksmith-testbox`, `syncDelegated=true`, and `tbx_...` ids. Both satisfy "remote proof" when the requested proof surface allows either. - Treat contributor and fork patches as untrusted unless a maintainer explicitly approves credentialed execution after review. For untrusted AWS runs, `CRABBOX_ENV_ALLOW=CI` must replace the repo's `OPENCLAW_*` allowlist, `--no-hydrate` must block auth-profile hydration, and the remote command must use a fresh temporary `HOME`. The lease must be newly warmed for and bound to one reviewed head SHA, never trusted or previously hydrated; stop and rewarm when the SHA changes. Do not execute repo scripts or config from the untrusted local checkout: launch an installed trusted Crabbox binary from a clean trusted `main` checkout and fetch the PR with `--fresh-pr`. Unset `CRABBOX_AWS_INSTANCE_PROFILE` and fail closed unless `crabbox config show --json` resolves an empty `aws.instanceProfile`. Before any install/test, use trusted absolute-path tools to require an IMDSv2 token, prove the IAM credentials endpoint returns 404, and compare remote `git rev-parse HEAD` with the full reviewed head SHA. Unset all `CRABBOX_TAILSCALE*` overrides, pass `--network public --tailscale=false`, clear exit-node/LAN flags, then require `crabbox inspect` to report `network=public` and no Tailscale state before uploading any script. Upload trusted `scripts/crabbox-untrusted-bootstrap.sh` with `--fresh-pr`; it bootstraps Node 24 and repository-pinned pnpm before executing PR code and rejects a changed `packageManager` pin before install. If the broker cannot provide that no-role proof or no remote PR exists, use secretless fork CI. Do not select `hydrate-github` or a credential-hydrated Testbox workflow. - Do not infer "no Testbox is running" from plain `blacksmith testbox list`. Use `blacksmith testbox list --all` or `blacksmith testbox status --id <tbx_id>` (id is not positional) before reporting cloud state. - Reuse only an id/slug created in this operator session unless explicitly coordinating with another lane. If Testbox queues, fails capacity, or cannot allocate, report the blocker or switch to direct AWS Crabbox only when that still proves the requested surface. - Blacksmith Testbox owns sync, including reused `--id` runs. Do not pass `--no-sync`; the wrapper rejects it before delegation. Verify the materialized candidate tree before exact-source proof and keep QA evidence outside the synced checkout. Do not bypass security exclusions or silently change providers. ## Local Development Proof Use the smallest command that proves the touched contract, then broaden locally when the risk requires it. Select a remote backend only for environment or isolation proof. ```bash pnpm changed:lanes --json pnpm check:changed # changed checks; may include targeted Vitest owner tests pnpm test:changed # cheap smart changed Vitest targets pnpm verify # full check, then full Vitest OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed pnpm test <path-or-filter> -- --reporter=verbose OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test <path-or-filter> ``` Independent `pnpm test`/Vitest runs and checks that schedule Vitest (including `pnpm check:changed`) must not share a cache when run concurrently in one worktree; cache races can fail with `ENOTEMPTY`. Group tests into one command, serialize test/check runs, or give each concurrent command a distinct `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH` value. Use targeted file paths whenever possible. Avoid raw `vitest`; use the repo `pnpm test` wrapper so project routing, workers, and setup stay correct. If raw Vitest is unavoidable, use `vitest run ...`; bare `vitest ...` starts local watch mode and will not exit on its own. In a linked worktree, use the direct Node harness when avoiding pnpm dependency reconciliation is useful: ```bash node scripts/run-vitest.mjs <path-or-filter> ``` That keeps the test scoped without giving pnpm a chance to run dependency status checks or install reconciliation in a linked worktree. ## Plugin Package And Live Proof When validating an external or official plugin package, prove the package shape and trust shape separately. Do not use raw archive/path installs to prove the managed dependency path, and do not treat `npm-pack:` as proof of catalog-linked official trust. - For local release-candidate proof, pack the plugin and install it with `openclaw plugins install npm-pack:<path.tgz> --force`. This uses the managed per-plugin npm project and is the closest local substitute for the registry artifact's dependency behavior. - If the behavior depends on bundled-plugin or trusted official plugin status, add a second proof through a catalog-backed official install or a published package path that records official trust. Local `npm-pack:` proof alone is not sufficient for privileged helpers or trusted-official scope handling. - Treat missing runtime imports as package-manifest bugs first. Runtime code must depend on packages declared in the plugin package `dependencies` or `optionalDependencies`; do not make a final proof depend on manually running `npm install` inside `~/.openclaw/npm/projects/...`. - After moving dependencies between dev and runtime sections, run the transient npm package-lock check and inspect the bundled runtime payload when enabled. - Inspect the packed tarball when dependency ownership or generated `dist/` matters: verify `package/package.json`, the expected runtime files, the bundled `node_modules` payload when enabled, and the absence of npm lockfiles before installing it on a live host. - After installing the package, restart the Gateway when the touched surface is plugin registration, runtime dependency loading, privileged helpers, provider routing, or generated dist. - For live provider or channel probes, add only temporary config needed for the proof, then remove it and verify the cleanup state before closeout. ## Command Semantics - `pnpm check` runs the aggregate formatting, typecheck, lint, and guard graph. - `pnpm check:changed` runs changed-scope checks and can also run targeted Vitest owner tests via `pnpm test:serial`. Non-test plugin modules or manifests trigger the doctor-contract declaration and closure-guard tests; prompt-snapshot, runtime-sidecar, and appcast changes also have owner-test branches. Inspect the actual plan with `node scripts/check-changed.mjs --dry-run -- <paths...>`. This is targeted owner coverage, not the full Vitest suite. - `pnpm test` and `pnpm test:changed` run Vitest tests. - `pnpm verify` runs `pnpm check`, then `pnpm test`, with Crabbox phase markers so remote summaries show which half failed. - `pnpm test:changed` is intentionally cheap by default: direct test edits, sibling tests, explicit source mappings, and import-graph dependents. - `OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed` is the explicit broad fallback for harness/config/package edits that genuinely need it. - Do not run extension sweeps just because core changed. If a core edit is for a specific plugin bug, run that plugin's tests explicitly. If a public SDK or contract change needs consumer proof, choose the smallest representative plugin/contract tests first, then broaden only when the risk justifies it. - The test wrapper prints a short `[test] passed|failed|skipped ... in ...` line. Vitest's own duration is still the per-shard detail. ## Routing Model - `pnpm changed:lanes --json` answers "which check lanes does this diff touch?" It is used by `pnpm check:changed` for typecheck/lint/guard selection. - `pnpm test:changed` answers "which Vitest targets are worth running now?" It uses the same changed path list, but applies a cheaper test-target resolver. - Direct test edits run themselves. Source edits prefer explicit mappings, sibling `*.test.ts`, then import-graph dependents. Shared harness/config/root edits are skipped by default unless they have precise mapped tests. - Shared group-room delivery config and source-reply prompt edits are precise mapped tests: they run the core auto-reply regressions plus Discord and Slack delivery tests so cross-channel default changes fail before a PR push. - Public SDK or contract edits do not automatically run every plugin test. `check:changed` proves extension type contracts; the agent chooses the smallest plugin/contract Vitest proof that matches the actual risk. - Use `OPENCLAW_TEST_CHANGED_BROAD=1 pnpm test:changed` only when a harness, config, package, or unknown-root edit really needs the broad Vitest fallback. ## CI Debugging Start with current run state, not logs for everything: ```bash gh run list --branch main --limit 10 gh run view <run-id> --json status,conclusion,headSha,url,jobs gh run view <run-id> --job <job-id> --log ``` - Check exact SHA. Ignore newer unrelated `main` unless asked. - For cancelled same-branch runs, confirm whether a newer run superseded it. - Fetch full logs only for failed or relevant jobs. - Prefer `gh run view <run-id> --json jobs` over PR rollup while debugging; rollup can be stale/noisy. - For `prompt:snapshots:check` failures, treat Linux Node 24 as CI truth. If macOS passes but CI drifts, reproduce in a Linux Node 24 container or Testbox, commit that generated output, then rerun. ## GitHub Release Workflows Use the smallest workflow that proves the current risk. The full umbrella is available, but it is usually the last step after narrower proof, not the first rerun after a focused patch. ### Full Release Validation `Full Release Validation` (`.github/workflows/full-release-validation.yml`) is the manual product-validation umbrella. Bind each run to the immutable **Validation SHA + Tooling SHA** tuple. Validation SHA maps to the Code SHA for product validation or the Release SHA for changelog-only validation; it is not a third release identity. The workflow resolves it before child dispatch, then dispatches: - manual `CI` for the full normal CI graph, with Android enabled via `include_android=true` - `Plugin Prerelease` for release-only plugin static checks, extension shards, the release-only `agentic-plugins` shard, and plugin product Docker lanes - `OpenClaw Release Checks` for install smoke, cross-OS release checks, package acceptance, and QA parity; broad live/E2E and QA-live lanes join `all` only when release soak is enabled - optional post-publish Telegram E2E when a package spec is supplied For beta-publish, use `release_profile=beta` with `run_release_soak=false`. Postpublish-confidence uses the exact published package with `run_release_soak=true` or explicit focused groups. Stable-publish uses `release_profile=stable`. ```bash TOOLING_SHA="<recorded-full-main-ancestor-sha>" node scripts/full-release-validation-at-sha.mjs \ --sha <code-sha> \ --target-ref release/YYYY.M.PATCH \ --workflow-sha "$TOOLING_SHA" ``` That helper is for regular releases. Extended-stable dispatches Full Release Validation directly from and against `extended-stable/YYYY.M.33` with `release_profile=stable`; its exact branch-tip evidence is fresh and cannot be replaced by a `release-ci/*` run. Use `$release-openclaw-ci` for its failure classification and run-identity rules. The helper verifies and pins the recorded Tooling SHA on trusted `main`, passes the resolved Code SHA as `expected_sha`, and records the canonical release branch as context. Reuse that SHA for the release; never refresh it from moving `main`. Regular release branches accept only their final package version or a matching beta prerelease. Tideclaw alpha validation uses its matching alpha branch and exact alpha tag. The helper infers `beta` for beta candidates and exact alpha tags, and `stable` for stable/correction versions. Pass `-f release_profile=full` only for the broad advisory provider/media sweep. Do not make `full` faster by silently dropping suites; use the bounded phase that matches the release decision. Standalone manual `CI` dispatches do not run the plugin prerelease suite, the extension batch sweep, or the release-only `agentic-plugins` Vitest shard. Those lanes are intentionally reserved for the separate `Plugin Prerelease` child so PRs, main pushes, and ad hoc broad CI checks do not spend Docker/package time or all-plugin runtime time on release-only product coverage. `Plugin Prerelease` performs a supplemental scan of checked-in npm package input as inert data; it never runs candidate lifecycle, asset, build, install, or replacement scanner code. This scan does not approve post-build or publication bytes. A future publisher redesign must scan the exact final bytes and publish that identical digest before this can become a publication gate; the current publisher does not provide that guarantee. Ingestion stays fail-slow so one malformed package cannot hide other package reports. Use one operator, one foreground owner, and at most one investigator for the current failed surface. Do not start `release-ci-summary --watch` while the SHA-pinned helper is already watching the same parent. Parent timeout or cancellation leaves adopted exact children running; cancel an exact child only by explicit operator action or by `fail_fast=true` after Release Decision binds the failure to that exact active run. The child-dispatch jobs record run ID, run attempt, and URL, then finish. The parent seals those tuples, original dispatch titles, gate coverage, reuse policy, and original parent attempt in one immutable `full-release-execution-plan-<run-id>` artifact and exact run-ID cache entry. Collector retries restore the cached bytes, validate them, and re-upload the artifact for their attempt before adopting its children; they never reconstruct the plan or redispatch tests. `Release Decision` polls those exact identities and can report `blocked_diagnostics_running` before unrelated children finish. For reused evidence, it also repeats the canonical target, policy, changed-path, selected-run, root-run, and exact-child validation before it can pass. `Diagnostic Drain` continues every selected child to terminal with `fail_fast=false` unless the collector itself is cancelled or loses API access. `orchestration_error` permits collector recovery against the same exact children, never test redispatch. Diagnose `blocked_diagnostics_running` immediately, but wait for a terminal drain before retrying the failed surface. The final `Verify full validation` job consumes and validates the immutable execution plan plus the exact Decision and Drain artifacts instead of reclassifying child results. A later narrow green run is useful recovery evidence but is not publish authorization by itself and there is no standalone finalizer. The release owner must reassess the recorded evidence and current publish gate. Once the Code SHA is green, generate and commit only `CHANGELOG.md`. The new **Release SHA** is eligible for product-evidence reuse only when GitHub proves that it is a descendant of the Code SHA and the complete changed path set is exactly `CHANGELOG.md`. Dispatch the same SHA-pinned helper for the Release SHA; the resulting parent records `changelog-only-release-v1` and reuses the Code SHA children. Package, install/update, and release-note proof still runs on the Release SHA because its tarball bytes changed. Any non-changelog path invalidates reuse and requires a new Code SHA full matrix. For bounded recovery, classify the failure as product, harness/tooling/provenance, infrastructure/credential, or wrapper before editing. Only a confirmed product failure changes the Code SHA. Use one diagnosis, one fix when needed, and one narrow retry with `-f rerun_group=<group>`, then reassess. Supported umbrella groups are `all`, `ci`, `plugin-prerelease`, `install-smoke`, `cross-os`, `live-e2e`, `package`, `qa-parity`, `qa-live`, `npm-telegram`, and `performance`. The old `release-checks` aggregate retry handle is invalid because it silently selected every release-check lane. `qa` is a direct-child manual aggregate, not an umbrella/controller retry API. Use the narrowest concrete group that covers the failed box. Do not automatically dispatch `all` after a narrow retry. For a single failed live/E2E shard, use `-f rerun_group=live-e2e -f live_suite_filter=<suite_id>` so the Blacksmith workflow only spends setup and queue time on that suite. ### Release Evidence After release-candidate validation or before a release decision, record the important run ids in the public `openclaw/releases` evidence ledger. Use the manual `OpenClaw Release Evidence` (`openclaw-release-evidence.yml`) workflow there. It writes durable summaries under `evidence/<release-id>/` and commits: - `release-evidence.md` - `release-evidence.json` - `index.json`