openclaw-test-performance

# OpenClaw Test Performance Use evidence first. The goal is real `pnpm test`, plugin-suite, and plugin-inspector speed/RSS improvement with coverage intact, not runner tuning by guesswork. ## Workflow 1. Read the relevant local `AGENTS.md` files before editing: - `src/agents/AGENTS.md` for agent/import hotspots. - `src/channels/AGENTS.md` and `src/plugins/AGENTS.md` for plugin/channel laziness. - `src/gateway/AGENTS.md` for server lifecycle tests. - `test/helpers/AGENTS.md` and `src/channels/plugins/contracts/test-helpers/AGENTS.md` for shared contract helpers. - `src/infra/outbound/AGENTS.md` for outbound/media/action tests. 2. Establish a baseline before changing code: - Prefer `pnpm test:perf:groups --full-suite --allow-failures --output <file>` for full-suite ranking. - For bundled plugin breadth, run the smallest relevant `pnpm test:extensions:batch <plugin[,plugin...]>` or plugin-inspector command before jumping to the full extension sweep. - For a scoped hotspot use: `/usr/bin/time -l pnpm test <file-or-files> --maxWorkers=1 --reporter=verbose` - For import-heavy suspicion add: `OPENCLAW_VITEST_IMPORT_DURATIONS=1 OPENCLAW_VITEST_PRINT_IMPORT_BREAKDOWN=1`. 3. Separate wall/runner noise from real file cost: - Compare Vitest duration, test body timing, import breakdown, wall time, and max RSS. - Re-run single files when grouped/full-suite numbers look stale or noisy. - If a full-suite grouped run reports a lane failure but JSON says tests passed, capture that as harness/noise and verify the suspect file directly. 4. Pick the next attack by return and risk: - High return: one file/test dominates seconds or RSS and has a clear root. - High leverage: one plugin or SDK barrel causes every plugin-inspector or extension-batch run to load broad runtime. - Lower risk: static descriptors, target parsing, routing, auth bypass, setup hints, registry fixtures, or test server lifecycle. - Higher risk: real memory/runtime behavior, live providers, protocol contracts, or broad production refactors. 5. Fix the root cause, not the symptom: - Move static metadata/parsing into narrow helpers or lightweight artifacts reused by full runtime and fast paths. - Prefer dependency injection, loaded-plugin-only lookup, explicit fixtures, and pure helpers over broad mocks. - Reuse suite-level servers/clients when a fresh handshake is irrelevant. - Keep schedulers/background loops off unless the test proves scheduling. - In plugin paths, move static metadata into manifest/lightweight artifacts and keep runtime plugin loads behind explicit execution boundaries. 6. Preserve coverage shape: - Do not delete a slow integration proof unless the exact production composition is extracted into a named helper and tested. - Keep one cheap integration smoke when cross-component wiring matters. - State explicitly what incidental coverage was removed, if any. 7. Re-benchmark the same command after the change and compute seconds plus percent gain. 8. Update the running report when requested or when this thread is tracking one. Include before/after commands, artifacts, coverage notes, verification, and next attack order. 9. Stage the intended paths, commit with standard Git, and push when the user asked for commits/pushes. Stage only files touched for this attack. ## Plugin-Suite Workflow Use this section when perf work involves bundled plugins, plugin-inspector, SDK barrels, package-boundary tests, or extension suites. 1. Map the suite shape first: - source tests: `pnpm test extensions/<id>` or `pnpm test:extensions:batch <id>` - package boundaries: `pnpm run test:extensions:package-boundary:canary` and `pnpm run test:extensions:package-boundary:compile` - all bundled source tests: `pnpm test:extensions` - plugin import memory: `pnpm test:extensions:memory -- --json .artifacts/test-perf/extensions-memory.json` - plugin-inspector/report work: keep report primitives in `plugin-inspector`; keep wrappers thin and collect peak RSS when the command supports it. 2. Start narrow, then widen: - one plugin changed: run that plugin's tests and plugin-inspector slice. - SDK/public barrel changed: add representative provider, channel, memory, and feature plugins. - loader/runtime mirror changed: add package-boundary checks and build/package proof as needed. - unknown shared plugin behavior: run `test:extensions:batch` groups before `pnpm test:extensions`. 3. Treat plugin-inspector failures as product signals: - JSON must parse. - warnings/errors must be classified, not hidden. - runtime capture should be quiet and config-tolerant. - command output should include wall time, exit code, and peak RSS when available. 4. For broad or package-heavy plugin proof, use the current dedicated Linux worker when available. On a maintainer workstation, use Crabbox-backed Blacksmith Testbox: - `pnpm crabbox:run -- --provider blacksmith-testbox --timing-json -- OPENCLAW_TESTBOX=1 pnpm test:extensions:batch <ids>` - add `--keep`/`--id <id-or-slug>` only when several commands must share one warmed box; stop it with `pnpm crabbox:stop -- <id-or-slug>`. 5. If plugin performance is package-artifact sensitive, switch to `release-openclaw-plugin-testing` and Package Acceptance rather than trusting source-only timing. ## Metric Collection Collect at least one stable metric before and after. Prefer the same machine and same command. For Testbox comparisons, use the same `tbx_...` id when possible. | Metric | Use for | Preferred source | | --------------- | ---------------------------------- | --------------------------------------------------------------------------- | | wall time | user-visible suite cost | `/usr/bin/time -l`, test wrapper duration, Testbox run time | | Vitest duration | test body/import cost | Vitest output per file/shard | | import duration | broad barrel/runtime loads | `OPENCLAW_VITEST_IMPORT_DURATIONS=1` | | max RSS | memory pressure and OOM risk | `/usr/bin/time -l`, `pnpm test:extensions:memory`, wrapper memory summaries | | CPU/user/sys | CPU-bound vs wait-bound split | `/usr/bin/time -l` locally, Testbox job timing when local CPU is noisy | | heap evidence | real leak vs retained module graph | `openclaw-test-heap-leaks` workflow | Local scoped command with CPU/RSS: ```bash timeout 240 /usr/bin/time -l pnpm test <file> --maxWorkers=1 --reporter=verbose ``` Plugin import memory profile: ```bash pnpm build pnpm test:extensions:memory -- --top 20 --json .artifacts/test-perf/extensions-memory.json ``` Targeted plugin import memory: ```bash pnpm test:extensions:memory -- --extension discord --extension telegram --skip-combined ``` Heap/RSS escalation: ```bash pnpm test:perf:groups \ --config test/vitest/vitest.unit-fast.config.ts \ --allow-failures \ --output .artifacts/test-perf/unit-fast-memory.json pnpm test:perf:profile:runner -- \ --output-dir .artifacts/test-perf/vitest-runner-profile -- <file> ``` Use `openclaw-test-heap-leaks` when RSS keeps growing across intervals, workers OOM, or the suspect command has app-object retention. Do not call RSS growth a leak until snapshots or retainers support it. ## Common Root Causes - Full bundled channel/plugin runtime loaded for static data. - `getChannelPlugin()` fallback used when an already-loaded fixture or pure parser would suffice. - Broad `api.ts`, `runtime-api.ts`, `test-api.ts`, or plugin-sdk barrels pulled into hot tests. - SDK root aliases or package barrels pulling focused subpaths back into a broad plugin graph. - Plugin-inspector loading runtime code just to render metadata, reports, or CI policy scores. - Bundled plugin capture reusing real config/home state instead of synthetic, redacted, isolated state. - Partial-real mocks using `importActual()` around broad modules. - `vi.resetModules()` plus fresh imports in per-test loops. - Test plugin registry seeded in `beforeAll` while runtime state resets in `afterEach`. - Per-test gateway/server/client startup when state reset would suffice. - Runtime/default model/auth selection paid by idle snapshots or fixtures. - Plugin-owned media/action discovery triggered before checking whether args contain plugin-owned fields. - Parallel Vitest runs sharing `node_modules/.experimental-vitest-cache` without distinct `OPENCLAW_VITEST_FS_MODULE_CACHE_PATH` values. ## Benchmark Commands Scoped file: ```bash timeout 240 /usr/bin/time -l pnpm test <file> --maxWorkers=1 --reporter=verbose ``` Scoped file with import breakdown: ```bash timeout 240 /usr/bin/time -l env \ OPENCLAW_VITEST_IMPORT_DURATIONS=1 \ OPENCLAW_VITEST_PRINT_IMPORT_BREAKDOWN=1 \ pnpm test <file> --maxWorkers=1 --reporter=verbose ``` Grouped suite: ```bash pnpm test:perf:groups --full-suite --allow-failures \ --output .artifacts/test-perf/<name>.json ``` Extension batch: ```bash pnpm test:extensions:batch <plugin[,plugin...]> -- --reporter=verbose ``` All extension tests: ```bash pnpm test:extensions ``` Package-boundary plugin checks: ```bash pnpm run test:extensions:package-boundary:canary pnpm run test:extensions:package-boundary:compile ``` Reuse an existing Vitest JSON report: ```bash pnpm test:perf:groups --report <vitest-json> \ --output .artifacts/test-perf/<name>.json ``` ## Verification - Always run the targeted test surface that proves the change. - For source changes, run `pnpm check:changed` before push; in maintainer Testbox mode run it in the warmed Testbox. - For test-only changes, run `pnpm test:changed` or the exact edited tests. - Run `pnpm build` when touching lazy-loading, bundled artifacts, package boundaries, dynamic imports, build output, or public surfaces. - For plugin SDK/barrel/runtime changes, compare exact commits with `pnpm plugin-sdk:api:diff -- --base <base-sha> --head <head-sha>` when the public API surface may drift. For PR-local proof, use the branch merge base as `<base-sha>` and the exact tested head commit as `<head-sha>`. - For plugin-suite perf fixes, verify at least one representative plugin batch plus the changed gate; use Package Acceptance if the bug only exists in a packed artifact. - If deps are missing/stale, run `pnpm install` and retry the exact failed command once. - Use the report format: ```markdown | Metric | Before | After | Gain | | -------------- | -----: | -----: | ------------: | | File wall time | `Xs` | `Ys` | `-Zs` (`P%`) | | Max RSS | `XMB` | `YMB` | `-ZMB` (`P%`) | | CPU user/sys | `X/Ys` | `A/Bs` | explain | ``` ## Handoff Keep the final concise: - Root cause. - Suite/plugin scope. - Files changed. - Before/after wall, Vitest/import, CPU, and RSS numbers where available. - Leak classification if memory was involved: real leak, retained module graph, or inconclusive. - Coverage retained. - Verification commands. - Testbox ID or workflow URL for remote proof. - Commit hash and push status.