| name | openclaw-qa-testing |
| description | Run, watch, debug, extend, or explain OpenClaw qa-lab and qa-channel scenarios, artifacts, and live lanes. |
OpenClaw QA Testing
Use this skill for qa-lab / qa-channel work. Repo-local QA only.
Read first
docs/concepts/qa-e2e-automation.mddocs/help/testing.mddocs/channels/qa-channel.mdqa/README.mdqa/scenarios/index.yamlextensions/qa-lab/src/suite.tsextensions/qa-lab/src/character-eval.ts
Model policy
- Normal live suite runs rely on QA Lab source- and auth-aware defaults.
- Do not pass
--model, --alt-model, or --fast by default. Omitted
--fast does not mean fast is disabled; fast behavior is source-owned.
- For scenario-specific runs, the complete
execution.summary is authoritative
and overrides generic default guidance, including when it requires other
flags. Add explicit provider/model pins only when
execution.config.requiredProvider or requiredModel requires them.
Default workflow
- Read the scenario pack and current suite implementation.
- Decide lane:
- mock/dev:
mock-openai
- real validation:
live-frontier
- For a normal live suite, use:
pnpm openclaw qa suite \
--provider-mode live-frontier \
--output-dir .artifacts/qa-e2e/run-all-live-frontier-<tag>
- Watch outputs:
- summary:
.artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-summary.json
- report:
.artifacts/qa-e2e/run-all-live-frontier-<tag>/qa-suite-report.md
- If the user wants to watch the live UI, find the current
openclaw-qa listen port and report http://127.0.0.1:<port>.
- If a scenario fails, fix the product or harness root cause, then rerun the full lane.
OTEL smoke
For local QA-lab OpenTelemetry validation, use:
pnpm qa:otel:smoke
This starts a local OTLP/HTTP trace receiver, runs the otel-trace-smoke
scenario through qa-channel, decodes the emitted protobuf spans, and verifies
the exported trace names and privacy contract. It does not require Opik,
Langfuse, or external collector credentials.
QA credentials and 1Password
- Use
op only inside tmux for QA secret lookup in this repo.
- Quick auth check inside tmux:
op account list
- Direct Telegram npm live test secrets currently live in 1Password item:
- vault:
OpenClaw
- item:
Telegram E2E
- That item is the first place to look for:
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKENOPENCLAW_QA_TELEGRAM_SUT_BOT_TOKENOPENCLAW_QA_PROVIDER_MODEOPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC
- Convex QA secrets currently live in 1Password items:
- vault:
OpenClaw
- item:
OPENCLAW_QA_CONVEX_SITE_URL
- item:
OPENCLAW_QA_CONVEX_SECRET_MAINTAINER
- item:
OPENCLAW_QA_CONVEX_SECRET_CI
- Additional related notes/login items seen during QA credential work:
- vault:
Private
- items:
OPENCLAW QA, Convex, Telegram
- If a required value is missing from those notes:
- do not guess
- ask the maintainer/operator for the current value or the current 1Password item name
- for Telegram direct runs,
OPENCLAW_QA_TELEGRAM_GROUP_ID may be stored separately from Telegram E2E
- for Convex runs, the leased Telegram credential should provide the Telegram group id and bot tokens together; do not require a separate
OPENCLAW_QA_TELEGRAM_GROUP_ID
- for Convex runs, prefer
OpenClaw/OPENCLAW_QA_CONVEX_SITE_URL; if that is stale or unclear, ask for the active pool URL before running
- Prefer direct Telegram envs for the npm Telegram Docker lane when available:
OPENCLAW_QA_TELEGRAM_GROUP_ID="..." \
OPENCLAW_QA_TELEGRAM_DRIVER_BOT_TOKEN="..." \
OPENCLAW_QA_TELEGRAM_SUT_BOT_TOKEN="..." \
OPENCLAW_QA_PROVIDER_MODE="mock-openai" \
OPENCLAW_NPM_TELEGRAM_PACKAGE_SPEC="openclaw@beta" \
pnpm test:docker:npm-telegram-live
- Prefer Convex mode when the goal is stable shared QA infra:
- round-robin credential leasing
- thinner wrapper for channel-specific setup
- CLI/admin flows around the pooled credentials
- Live npm Telegram Docker lane note:
scripts/e2e/npm-telegram-live-runner.ts reads OPENCLAW_NPM_TELEGRAM_PROVIDER_MODE- do not assume
OPENCLAW_QA_PROVIDER_MODE is consumed by that wrapper
- if a 1Password note only gives
OPENCLAW_QA_PROVIDER_MODE, map it explicitly to OPENCLAW_NPM_TELEGRAM_PROVIDER_MODE before running the Docker lane
- Verified live shape:
- Convex mode can pass the real Docker lane without direct Telegram env vars
- leased Telegram payload includes the group id coupled to the driver/SUT tokens
- a real run of
pnpm test:docker:npm-telegram-live passed with:
OPENCLAW_QA_CREDENTIAL_SOURCE=convexOPENCLAW_QA_CREDENTIAL_ROLE=maintainerOPENCLAW_QA_CONVEX_SITE_URLOPENCLAW_QA_CONVEX_SECRET_MAINTAINEROPENCLAW_NPM_TELEGRAM_PROVIDER_MODE=mock-openai
- If direct Telegram env is missing locally and
op signin blocks, prefer dispatching the manual GitHub lane because the qa-live-shared environment already has Convex CI credentials:
gh workflow run "NPM Telegram Beta E2E" --repo openclaw/openclaw --ref main \
-f package_spec=openclaw@YYYY.M.D-beta.N \
-f package_label=openclaw@YYYY.M.D-beta.N \
-f provider_mode=mock-openai
- Poll the exact run id from the dispatch URL.
gh run view --json artifacts is not supported; list artifacts with:
gh api repos/openclaw/openclaw/actions/runs/<run-id>/artifacts
WhatsApp live credentials
Use this when setting up or replacing Convex kind=whatsapp credentials.
- Treat WhatsApp QA credentials as operator-owned live accounts, not generated fixtures.
- Use two dedicated WhatsApp-capable test numbers: one driver account and one SUT account. Do not use personal numbers or personal OpenClaw WhatsApp accounts in the shared pool.
- Register and link each account manually with WhatsApp or WhatsApp Business, storing Web auth only in isolated local auth dirs outside the repo.
- For group coverage, create a dedicated test group that includes both QA accounts and store its JID as
groupJid; otherwise the group mention-gating scenario should be skipped by default and fail when explicitly requested.
- Package the two Baileys auth dirs into base64
.tgz payload fields and add a new active Convex credential row. Prefer adding a fresh row and disabling stale/broken rows over overwriting credentials in place.
- Expected payload fields:
driverPhoneE164, sutPhoneE164, driverAuthArchiveBase64, sutAuthArchiveBase64, and optional groupJid.
- Keep credential material out of the repo, logs, PRs, and screenshots. Redact phone numbers unless the operator explicitly asks for local debugging.
- Validate with
pnpm openclaw qa whatsapp --credential-source convex --credential-role maintainer --provider-mode mock-openai and preserve artifact paths plus redacted pass/fail summaries.
- If WhatsApp expires or invalidates a linked Web session, relink locally, package fresh auth archives, add a new Convex row, then disable the stale row.
Character evals
Use qa character-eval for style/persona/vibe checks across multiple live models.
pnpm openclaw qa character-eval \
--output-dir .artifacts/qa-e2e/character-eval-<tag>
- Runs local QA gateway child processes, not Docker.
- Packaged
pnpm build omits QA Lab + qa-channel by design (source-checkout
only). To exercise openclaw qa/qa-channel from a built dist, build with
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm build (emits dist/plugin-sdk/qa-lab.js,
qa-runtime.js, dist/extensions/{qa-lab,qa-channel}) or run via pnpm dev.
- With no model flags, character eval uses its current source-defined candidate,
judge, thinking, and fast defaults.
- Repeat
--model provider/model,thinking=<level>[,fast|,no-fast|,fast=<bool>]
or --judge-model ... only to replace the corresponding inventory explicitly.
- Do not add new examples with separate
--model-thinking; keep that flag as legacy compatibility only.
- Report includes judge ranking, run stats, durations, and full transcripts; do not include raw judge replies. Duration is benchmark context, not a grading signal.
- Candidate and judge concurrency default to 16. Use
--concurrency <n> and --judge-concurrency <n> to override when local gateways or provider limits need a gentler lane.
- Scenario source is YAML-only under
qa/scenarios/: use index.yaml and
per-scenario *.yaml files with top-level title, scenario, and optional
flow. Never add fenced qa-scenario / qa-flow Markdown files.
- For isolated character/persona evals, write the persona into
SOUL.md and blank IDENTITY.md in the scenario flow. Use SOUL.md + IDENTITY.md only when intentionally testing how the normal OpenClaw identity combines with the character.
- Keep prompts natural and task-shaped. The candidate model should receive character setup through
SOUL.md, then normal user turns such as chat, workspace help, and small file tasks; do not ask "how would you react?" or tell the model it is in an eval.
- Prefer at least one real task, such as creating or editing a tiny workspace artifact, so the transcript captures character under normal tool use instead of pure roleplay.
Codex CLI model lane
Use model refs shaped like codex-cli/<codex-model> whenever QA should exercise Codex as a model backend.
Examples:
pnpm openclaw qa suite \
--provider-mode live-frontier \
--model codex-cli/<codex-model> \
--alt-model codex-cli/<codex-model> \
--scenario <scenario-id> \
--output-dir .artifacts/qa-e2e/codex-<tag>
pnpm openclaw qa manual \
--model codex-cli/<codex-model> \
--message "Reply exactly: CODEX_OK"
- Treat the concrete Codex model name as user/config input; do not hardcode it in source, docs examples, or scenarios.
- Live QA preserves
CODEX_HOME so Codex CLI auth/config works while keeping HOME and OPENCLAW_HOME sandboxed.
- Mock QA should scrub
CODEX_HOME.
- If Codex returns fallback/auth text every turn, first check
CODEX_HOME,
relevant secret-backed auth, and gateway child logs before changing
scenario assertions.
- For model comparison, include
codex-cli/<codex-model> as another candidate in qa character-eval; the report should label it as an opaque model name.
Repo facts
- Seed scenarios live in
qa/scenarios/index.yaml and
qa/scenarios/<theme>/*.yaml.
- Main live runner:
extensions/qa-lab/src/suite.ts
- QA lab server:
extensions/qa-lab/src/lab-server.ts
- Child gateway harness:
extensions/qa-lab/src/gateway-child.ts
- Synthetic channel:
extensions/qa-channel/
What “done” looks like
- Full suite green for the requested lane.
- User gets:
- watch URL if applicable
- pass/fail counts
- artifact paths
- concise note on what was fixed
Common failure patterns
- Live timeout too short:
- widen live waits in
extensions/qa-lab/src/suite.ts
- Discovery cannot find repo files:
- point prompts at
repo/... inside seeded workspace
- Subagent proof too brittle:
- prefer stable final reply evidence over transient child-session listing
- Harness “rebuild” delay:
- dirty tree can trigger a pre-run build; expect that before ports appear
When adding scenarios
- Add or update scenario YAML under
qa/scenarios/; do not add .md scenario
files or fenced YAML blocks.
- Keep kickoff expectations in
qa/scenarios/index.yaml aligned
- Add executable coverage in
extensions/qa-lab/src/suite.ts
- Prefer end-to-end assertions over mock-only checks
- Save outputs under
.artifacts/qa-e2e/
