怎么测 / 测试方案 / 测试覆盖 / 先写测试用例（可执行测试代码/覆盖） / test cases first for test code / 写测试 / 写测试代码 / 补测试 / 补测试用例（未指向文档/Bitable/source-to-TC 时按可执行测试覆盖处理） / mock / 单元测试 / 集成测试 / E2E / 回归 / CI gate → choose test layers, scenarios, fixtures, mocks, regression coverage, smoke, verification evidence, and runner-config policy. Skip structured test-case documents, Feishu/Bitable TC sync, or source-to-TC generation → test-artifact-management. Skip 具体的 runner 调用（pytest / go test 命令与参数怎么写）→ 该技术栈 dev 技能；覆盖阈值与 CI gate 策略仍归本技能。
Testing Strategy
Use this skill when the task is about test scope, test design, verification evidence, CI gates, or deciding which layer should prove behavior. It is stack-agnostic: apply it across Go services, Python services, React web, mobile apps, mini-programs, terminal/CLI/TUI surfaces, APIs, workflows, data/inference features, and product releases.
Skill Routing
Use this skill for test strategy, test pyramid decisions, unit/integration/e2e/scenario boundaries, fixtures, mocks/fakes, regression coverage, and verification evidence.
Use go-microservice-dev for Go-specific implementation details after the test layer is chosen.
Use python-service-dev for Python-specific pytest, async test, fixture, fake client, ruff, mypy/pyright, packaging, or migration-test implementation details after the test layer is chosen.
Use app-cross-platform-dev for Flutter unit/widget/integration tests, Android unit/Compose/instrumented tests, iOS unit/UI tests, mobile device smoke, and app release verification after the test layer is chosen.
Use miniapp-product-dev for WeChat/Alipay/Douyin/Baidu mini-program build, developer-tool preview, real-device smoke, host capability verification, review submission, and release checks after the test layer is chosen.
Use web-react-dev for React unit/component/API-client/browser/E2E implementation, browser automation, accessibility checks, and build/deploy smoke after the test layer is chosen.
Use terminal-cli-dev for command-line, terminal, PTY-backed, ANSI-rendered, keyboard-driven, and full-screen text UI implementation after the test layer is chosen.
Use llm-inference-integration for LLM/agent/RAG evals, replay sets, prompt/model regression checks, shadow runs, and inference-specific observability after the test layer is chosen.
Use defect-diagnosis first when a test is failing, flaky, or used to reproduce a bug.
Use product-rd-workflow first when testing is part of an end-to-end product delivery plan.
Use product-ui-ux-design for visual hierarchy, interaction, state, accessibility, and design acceptance criteria; use this skill for automated proof and test coverage.
Use test-artifact-management when the ask is about generating structured test cases from a Feishu requirements doc or codebase and delivering them to Feishu Bitable; use this skill for scenario selection and risk matrix before test-artifact-management writes the cases.
For ambiguous "写测试用例" / "write test cases" requests, use this skill first when the user is asking to add, modify, run, or verify test code, E2E/browser/device tests, CI coverage, mocks, or regression evidence. Route to test-artifact-management only when the deliverable is a structured TC artifact, Feishu/Bitable sync, or source-to-TC generation workflow.
For team testing standards, QA norms, CI/harness requirements, or a missing "测试规范" in a product/R&D standards family, use references/rd-testing-standard-template.md as the source-neutral template, then route stack-specific command mechanics to the relevant dev skill.
Non-Functional And Specialized Testing
Use this skill to decide whether a specialized or non-functional test belongs in a gate and what evidence is required; the domain owner still owns implementation semantics and detailed method. Keep the entrypoint as the ownership/pointer surface. The full domain index, per-domain gate criteria, scenario/assertion matrices, and routing handoffs live in references/non-functional-specialized-scenarios.md.
Core Rules
Test behavior and contracts, not implementation trivia.
Code analysis may add risk scenarios; it must not define product-delivery completeness. Derive primary coverage from the active requirement/acceptance source with stable IDs.
Do not write change-detector tests — assertions that snapshot data expected to change (vendor/catalog entries, config-version literals, enumeration counts, hardcoded registry lists) instead of behavior; they add no coverage and break CI on routine data updates. Reviewers convert them to relationship invariants plus a minimal product-guaranteed anchor, never transient values. Keep a literal only when it traces to a documented product/API/security contract or past defect, not implementation state, and the test name states that contract; otherwise, after confirming no behavior coverage is lost, convert it to a relationship invariant. Heuristic, exceptions, and examples: references/test-code-authoring-patterns.md §3.
Every test should pay for itself: prioritize by business risk, defect history, change frequency, and manual-verification cost.
Reuse before generating. Before writing any test, grep existing test files for coverage of the same function, endpoint, class, or scenario. If coverage exists, extend or parameterize it instead of creating a new test function or file. Duplicate test structure wastes maintenance and produces misleading coverage numbers.
When the change CORRECTS a behavior (a compliance, contract, privacy, or spec-conformance fix), that same grep has a second job: find the assertions that PIN THE OLD, NOW-WRONG BEHAVIOR — and sweep every layer that asserts it, not only the unit tests you are already editing. Rewrite each pinned assertion to the corrected contract in the same change (never delete it to go green; the corrected form is what stops the old behavior from returning). Why the old behavior is always pinned somewhere, and how to enumerate the layers to sweep: references/test-code-authoring-patterns.md §10.
Keep default tests fast, deterministic, and offline. Select local evidence by touched surface. Leave the matrix to CI only if verified blocking jobs cover it; otherwise run the missing gate or mark incomplete. Report commands run; reuse green only on the same recorded tree (references/ci-fixtures-and-flake-control.md, Local Evidence Selection).
Do not infer coverage from test file count or names. Classify tests by the command that runs them, marker/tag, required dependencies, CI placement, assertion strength, and whether failures block delivery. An uncovered line is first a delete candidate, then a test candidate (references/test-code-authoring-patterns.md §6).
Classify test evidence quality before using it in a completion claim: strong tests are deterministic and assertion-based; medium tests assert behavior but depend on fixed DB/cache/network state; weak tests only log/print, sleep, use random/live data, call live model services, or sit behind non-blocking CI. Weak tests can inform risk but cannot prove correctness. Evidence about an agent's own output observes the world, not its self-report (, Verify The World).
Entry Decision: TC Source and Scope
Before any test design or implementation work, resolve two questions: where do the test cases come from, and what is the scope.
Step A — Resolve acceptance and TC sources
For behavior-changing product delivery, always resolve the acceptance source first. Record acceptance-source lookup: found <artifact/section> or acceptance-source lookup: absent <locations checked>. When found, preserve the complete point-ID set and source decisions; TCs may refine but not silently narrow it. When absent, functional completeness is blocked: route to product-rd-workflow to establish the acceptance inventory — test-artifact-management may only run after an authoritative inventory exists, because it can generate cases from existing code and would reproduce the omission blind spot; code analysis remains supplemental. A delivery carrying an entry-classified functional-axis exemption claim (documentation-only, pure refactor, or mechanical maintenance per product-rd-workflow; the reviewer confirms or rejects the claim at closeout) is outside this step — cover it as existing-behavior/refactor coverage under step 4.
User provided TC URL or file path → fetch and use it directly.
.report-config.json exists in the working directory → auto-read base_token + table_id; fetch active TCs from Bitable via lark-cli base +record-list; filter out 废弃 records before passing to the implementation queue. Do not ask the user — auto-read is the default.
No TC source found → code analysis may define existing-behavior or refactor coverage. For product delivery, continue from the acceptance source, record the TC gap, and use code-derived scenarios only as supplemental risk coverage.
When TCs are available, pass the full list through When Input Is a Structured TC List and references/structured-tc-input-translation.md before doing anything else.
TC coverage vs. test code coverage — expected gap:
TCs cover functional and interaction scope (QA perspective: user journeys, acceptance criteria, error paths, permissions). Test code covers a broader surface: unit-level logic, internal invariants, boundary conditions, data validation, refactoring safety nets. These two scopes do not and need not align 1:1.
Test code without a TC ID is normal and expected. It runs as part of code quality coverage and does not sync to Bitable. Do not force-add TC IDs to tests that have no corresponding Bitable entry — that creates phantom TC links. When the underlying code is removed or deprecated, these tests should be reviewed and the user prompted to decide whether to delete them.
A TC entry without automated test code is also valid: manual test, deferred automation, or blocked environment.
The automated report (gen_report.py) tracks TC-mapped results (tests that register TC IDs via the tc(...) helper — see test-artifact-management/references/tc-marker-conventions.md) plus a separate "未链接 TC 的测试" section for tests without TC links. Broader code coverage is a code quality concern tracked separately (e.g. coverage reports, CI pass/fail).
Step B — Determine scope (if not already bounded)
A current acceptance source or reviewable task artifact may already confirm scope. In particular, a UI/UX Design brief with a stable slice/surface, authoritative consumer inventory, affected owner(s), and criterion IDs is the confirmed bounded scope for its Phase 0/Phase 1 work; consume it instead of asking the user to restate global/file/function scope.
If neither the request nor a current authoritative artifact bounds the work, inspect the current task, repository contract, diff/target, and available acceptance sources first. Ask only when two or more plausible scopes remain and choosing among them would materially change the test plan. Then ask the smallest concrete question, for example:
请确认测试范围：
全局 — 整个仓库 / 当前 feature 所有文件
文件 — 指定文件（请提供路径）
函数 / 接口 — 指定函数或 API endpoint（请提供名称）
Do not infer scope from stale conversation or an unverified guess. A current, resolvable Design brief or accepted scope artifact is evidence, not inference. If the inspection leaves one material scope, proceed and record its source; if ambiguity remains, wait for the user's answer before layer assignment.
When scope is confirmed:
全局: run the full scenario matrix from testing-strategy workflow; consult TC list for all active TCs.
文件: narrow TC list to TCs whose 模块 or 功能点 maps to that file; run scenario matrix for that file's contracts and business rules.
函数 / 接口: narrow to TCs directly covering that function or endpoint; add negative-path and boundary cases if not already in the TC list.
After resolving TC source and scope, proceed to When Input Is a Structured TC List and references/structured-tc-input-translation.md (if TCs available) or Workflow Step 1 (if no TCs).
When Input Is a Structured TC List
When the input is a set of structured test cases from test-artifact-management or a Feishu Bitable TC list, use references/structured-tc-input-translation.md as the translation guide, then run the standard workflow from Step 1. The mapping is directional — not every TC becomes one automated test, and one TC may need assertions at multiple layers.
Keep these entrypoint anchors in view while translating: normalize TC status before implementation (full status table in the reference); treat 失败 as a regression candidate requiring RED-before-implementation or a named evidence gap; treat 阻塞 as blocked with owner/gap; exclude 跳过 / 废弃 from the implementation queue without deleting or overwriting Bitable records; treat P0/P1/P2 as business risk signals, not direct layer or CI-gate assignments; keep TC IDs in sidecar/helper metadata rather than test function names; let this skill choose the layer/gate, stack skills implement mechanics, and test-artifact-management own TC content changes and Bitable sync. After pre-check and field mapping, run the standard workflow from Step 1 (discover repository test topology).
Workflow
Before editing tests, CI gates, mocks/fakes, fixtures, test scripts, verification docs, or implementation tied to a testing request, complete enough analysis and planning for the testing change to be reviewable. Scale the plan to risk: a simple low-risk single-test addition can use a short inline plan; multi-layer, user-visible, contract-visible, flaky, release-blocking, bug-regression, branch/MR, unclear-risk, or high-risk work needs explicit scenario/risk matrix, test-case register, task split, expected failing test or evidence gap, verification commands, and stop conditions before edits.
Discover the repository's test topology.
Read local guidance first: AGENTS.md, CLAUDE.md, README, CONTRIBUTING, local agent skill/instruction directories, Makefile, package.json, pyproject.toml, go.mod, pytest.ini, vitest/playwright config, CI jobs, scripts/run_tests.*, and scripts/dev.*.
Identify default fast tests, opt-in integration markers, contract/architecture checks, browser/E2E suites, local container stacks, and release smoke commands.
Exclude dependency and generated-test noise from the topology: node_modules, virtualenvs, vendored/third-party trees, generated SDK snapshots, build outputs, and archived worktrees unless the task is explicitly about those artifacts.
Prefer documented wrappers over raw runners unless the task is to debug the runner itself.
If a frontend or app repository has only build/dev/format scripts and no assertion-based UI tests, do not treat that as adequate coverage for interaction changes. Use build/typecheck as a structural gate, then add focused state/API tests or record the missing test layer and require rendered browser/device evidence.
If CI mainly builds images or deploys by branch/environment, classify it as release plumbing. It does not prove product behavior unless assertion-based test jobs run and block delivery.
When adding a new client platform scaffold, inspect whether the generator added default sample tests. Keep a sample only if it asserts real product behavior; otherwise replace it with the smallest deterministic smoke that would fail if the current shell, navigation, entrypoint, or visible state were missing.
Identify the behavior to prove.
User/caller outcome.
Contract, state, side effect, emitted event, rendered UI, permission behavior, or operational signal.
Risk if wrong.
Scenario dimensions when relevant: persona/role, entry path, data state, permission, dependency health, device/browser/platform, locale/timezone, network quality, and recovery path.
For API contract changes, first record the contract definition being tested: request shape, response envelope, success/error semantics, compatibility mode, consumers, and rollout assumption. If this is missing, stop and route back to product/architecture before writing broad tests.
For protobuf-backed HTTP, JSON/OpenAPI vs protobuf wire-format, and internal RPC/base/generated DTO paths, use the canonical policy/proof references from : , , and . This skill owns assertion coverage, verdict shape, and CI placement; platform-service-connectivity owns the platform contract semantics and proof mechanics.
Reference Loading
For source provenance, current extraction boundary, and keep/merge/discard decisions, read references/source-evidence-map.md when auditing or re-extracting this skill.
For discovering repo-specific commands, test directories, wrappers, and CI placement, read references/test-topology-and-commands.md.
For scenario matrices, user journeys, acceptance coverage, and mapping scenarios to test layers, read references/scenario-testing.md.
For unit tests, fakes, table tests, and deterministic fast targets, read references/unit-testing.md.
For test data, factories, fixtures, fuzzing, property tests, and deterministic controls, read references/test-data-and-determinism.md.
For API, contract, DB, Redis, MQ, generated code, and dependency integration tests, read references/integration-contract-testing.md.
For batch jobs, ETL/ELT, streams, scheduled jobs, backfills, replay, and data-quality assertions, read references/data-and-workflow-testing.md.
For browser, API workflow, real-flow, release smoke, and E2E evidence, read references/e2e-real-flow-testing.md.
For CI gates, test data, fixtures, flake control, and verification reporting, read references/ci-fixtures-and-flake-control.md.
For translating structured TC lists from test-artifact-management or Feishu Bitable into test-layer choices, status handling, TC ID metadata, execution-type routing, and handoff rules, read references/structured-tc-input-translation.md.
For how to write the test code itself — the 8 stack-agnostic authoring patterns (structure, naming, smells, fixtures, behavior-vs-state, coverage, isolation, parameterization) plus the closeout checklist — read references/test-code-authoring-patterns.md (steps 4–5 route here).
For architecture fitness functions (Ford et al. 2017) — automated tests that verify architectural invariants (no circular deps / DAL-only DB access / latency budget / module-boundary enforcement / breaking-change detection / security invariants), read . Different from product correctness tests; protects structural commitments declared in ADRs from drift.
references/e2e-real-flow-testing.md
Treat scenario testing as a planning layer, not automatically as E2E: start from user/caller journey, risk, persona/role, data state, environment, and expected evidence, then place assertions at the unit/contract/integration/E2E/exploratory/release-smoke layer.
Every critical scenario needs at least one automated assertion at the lowest layer that can prove the risk, plus real-flow smoke only when cross-boundary behavior matters; built/installed deliverables need one smoke via the published entry path (references/e2e-real-flow-testing.md, Published Entry Path).
Happy-path tests are insufficient for high-risk workflows. Build a compact risk matrix and cover the triggered failure classes — the canonical failure-class list lives in references/scenario-testing.md.
Composite / multi-stage pipelines (a capability built from several modules, services, or model stages chained together) need acceptance at three layers, not just one: module-level (the changed stage's own correctness, errors, latency, fallback), chain-level (upstream/downstream input-output contracts, version pass-through, end-to-end recovery and rollback), and product-level (the user-visible outcome / acceptance baseline holds). A change to one sub-module cannot pass on its local metric alone if the end-to-end product result regresses; require an end-to-end check whenever the pipeline composition or a stage contract changes. Scope and the unchanged-contract refactor exemption: references/scenario-testing.md. (For AI/inference pipelines the component-vs-end-to-end split and "launch follows the product baseline, not the best component metric" rule are owned by llm-inference-integration; this rule is the stack-agnostic testing form.)
Maintain a frozen regression set of real past failures (every fixed bug / incident / confirmed bad case becomes a case) and run it on every release, not ad hoc. Tier it so the gate stays affordable, plus a periodic adversarial pass over code considered "done" (tiering and adversarial-pass procedure: references/ci-fixtures-and-flake-control.md). For AI features, the online-bad-case → regression-set re-injection mechanics are owned by llm-inference-integration; the release-blocking gate stays here.
Use the repository's own test wrapper when it exists; it encodes env, codegen, fixture, timeout, and CI parity.
Use the right test double: stub to supply inputs, fake to emulate dependency behavior, and spy/mock to verify interaction only when the interaction is the contract; double the expensive/nondeterministic/unsafe/privileged/unavailable boundary and keep the rest real on isolated test-owned resources (references/test-code-authoring-patterns.md §5; external-provider recovery paths: references/ci-fixtures-and-flake-control.md, Fault-Injection Layers).
Use integration tests where mocks would hide contract, transaction, serialization, permission, data-shape, or runtime failures.
Use E2E only for critical user/caller workflows and release confidence, not every branch through the browser.
A passing test must assert the outcome. Clicking a button, calling an endpoint, or seeing status 200 is not enough. Every property a test NAMES — in ANY name a reader or a runner sees: the function name, a docstring or comment, and equally an it(...)/describe(...) title, a Gherkin scenario name, a pytest parameter id, or a subtest/table-case label (..._is_bijective, ..._is_pinned, "rejects out-of-range input") — is a claim, and each named property owes a killing mutation: the concrete implementation change that would make this test fail.
Drive the walk from the named properties in every changed test artifact, not from the changed assertions — renaming a test or adding a docstring line introduces a new claim while touching no assertion at all. For each named property write the mutation: break what the property names → the test must go RED → restore and re-run. Mutate in a disposable checkout or with the restore guarded — never by hand-editing a dirty worktree.
Guard it positively rather than resolving to be careful, and keep the guard out of version control — the executable backup recipe, the git-as-scratch-buffer rebuttal, and the blast-radius discipline live in references/run-killing-mutation-walk.md. Verify the tree is back to its pre-mutation state before trusting any later run, since a green result on a still-mutated tree certifies the wrong thing.
When the artifact under test is DESTRUCTIVE or irreversible (a script/recipe that deletes/overwrites/prunes worktrees, branches, files, records; bulk or --force-class mutation), a one-off walk is not enough to discharge this rule: encode the walk as a probe inside the suite — a bare non-zero exit does not qualify; right-reason/differential attribution and the probe's anchor/recursion guards live in references/run-killing-mutation-walk.md.
The walk proves only the pins that exist — coverage runs in the reverse direction, artifact → pin: an unpinned obligation produces no signal in any walk and is silently deletable. Every obligation sentence of the pinned artifact must name the pin that reds on its deletion; enumeration method and structural probes live in references/run-killing-mutation-walk.md.
A property whose killing mutation you cannot name is not asserted — add the assertion, or rename the test to what it checks.
Two shapes reading the test cannot catch: a test asserting only that a constant equals its own literal under a behavioral name (a change-detector per the rule above), and — the narrow case — a reverse expectation computed from the same source as the forward one, so both drift together and the assertion cannot disagree with itself; the round-trip nuance lives in references/run-killing-mutation-walk.md.
For nondeterministic output, assert structure, invariants, bounds, and required content rather than exact text. When the same contract has two implementations or paths (migration, port, rewrite, dual-path rollout), prefer differential/equivalence tests — feed the same input through both and assert agreement — over per-path golden snapshots; any exemption/known-difference list must carry an asserted size cap so exemptions cannot grow silently until the protection is gone.
Assert a security/safety boundary positively when the contract is closed (only-X-is-allowed): prove the observed surface matches the allowed contract, not merely that known-bad values are absent. A denylist of forbidden values is open-ended, so it passes while the boundary is actually violated — a false green. Mirror the documented contract with an independent oracle; don't snapshot a generated/runtime allowlist (that becomes a change-detector). Negative/adversarial cases stay valid — they prove out-of-contract input is rejected. Acting signal: when an adversarial reviewer keeps surfacing a new instance of the same bypass class across rounds, prefer an allowlist/invariant that closes the class over banning one more value, and prove it with a mutation where feasible (a boundary-violating change must make the test fail). The denylist failure shapes, the parsed-result-without-inputs gap, the dynamic-set fallback, and the documented-entry distinction live in references/design-closed-contract-oracles.md.
A clean oracle is clean only over the DIMENSIONS it crossed, not the cases it ran. A clean run must be reported with the dimensions it covers, and before trusting it you must enumerate the dimensions themselves and mark each crossed or not: the shape of a value; its provenance; cardinality; semantics; and ordering where the consumer is order-sensitive. The per-dimension explanations and the four-round failure shape live in references/design-closed-contract-oracles.md.
A guard's cases must pin PRECISION as well as recall — the false-positive direction is where hardening turns into an outage. Keep benign near-miss cases — inputs that share a prefix, substring, or shape with the violating class and must pass — as permanent rows next to the violating ones, so a later tightening that over-fires turns a row RED instead of shipping. The dual-track sweep relationship and the short-stem failure shape live in references/design-closed-contract-oracles.md.
For user-visible, contract-visible, bug-fix, or test-harness behavior changes, follow the test-case register and RED-before-implementation rule below; do not downgrade it to a preference.
Never delete, skip, or weaken a failing test just to pass — route failures through defect-diagnosis; removing a redundant/obsolete/duplicate test is valid maintenance, weakening or skipping a failing one is not.
When a CI step or review claim compares a checked-in copy of a vendored external contract/conformance artifact (vendored API/schema/wire/envelope/validator/fixture content — ambiguous upstream artifacts default in-scope; lockfile/digest-pin checks that vendor no such content are out) against a manifest, digest list, or pinned source, run references/vendored-contract-drift-checklist.md as the authoritative gate: enumerate scope → fill the per-row applicability ledger → emit an explicit pass/non-pass/inapplicable verdict (never a bare scope label or a comprehensive green). The load-bearing invariants are the checklist's own rows — absent-sibling fail-closed, upstream authenticity as a separate gate, single-canonical-producer recipe/wrapper that stays network-/delete-/exec-free beyond the canonical script with fetch/install only via a human-authorized sync gate, clean+RED-or-N/A per applicable row, RED only in scratch — applied from the checklist, not from memory. This is the test mechanic for the vendor-sync gate that product-rd-workflow cross-repo coordination routes here.
Any claim that the implementation AGREES WITH an external contract you do not own — pinned to, conforms to, compatible with, implements, supports, matches, or interoperates with a peer schema, published wire/file format, or vendor envelope; the trigger is the agreement claim, not one wording of it — requires at least one real sample produced by that side to have been exercised; fixtures hand-built from the contract document prove only that the code matches your reading of it. With none obtainable, record real-sample: unavailable and scope the claim to "matches the documented contract", never "matches the peer". A real sample is untrusted payload — screen and redact before committing it as a fixture. Method, per-cell coverage scoping, and downgrade wording: references/integration-contract-testing.md.
Benchmarks, smoke scripts, and manual click-throughs never replace assertion-based unit, contract, integration, or E2E tests; neither do structure scripts, scaffold checks, grep checks, or compile-only gates when a change adds real UI interaction, routing, API clients, permission behavior, failure handling, or user-visible state. If that is the repository's only test layer, add a small deterministic test framework and focused tests in the same delivery slice instead of treating the scaffold check as enough.
For UI that calls APIs, one test layer is not enough. Cover the page at three boundaries when feasible: component/unit tests for local state and rendered states, API-client/contract tests for response parsing and error translation, and browser/E2E smoke for at least one real critical path plus one important failure path.
Mocked component tests must not be treated as proof that runtime integration works. If all mocks return ideal JSON, add negative-path client tests for empty body, invalid JSON, non-2xx envelopes, network failure, timeout/abort where relevant, and permission/feature-closed responses.
A test that pins an input the production default leaves unset — or supplies a controlled value for a normally-defaulted condition — can pass while the real default/unset path is broken. When the pinned input substitutes for a production default path that could mask caller behavior, also exercise the unset/default path at the lowest layer that actually resolves that default (a parameterized unit test suffices when the default is local; use the real command/demo only when the default is assembled across runtime/caller boundaries). For a helper that retains mutable state, caches, accumulates, or reuses a process/global resource, add a multi-call run when repeated calls can affect the observable contract, failure mode, resource lifetime, or caller-visible state (not when repeated calls are intentionally identical and only performance differs). A green test over only the convenient pinned input is not proof the default path works. (Tests that intentionally pin impossible/legacy/migration-only inputs are exempt.)
When an enforcement gate, tripwire, or guarantee is re-routed through a new indirection layer (A→B→C instead of A→C) AND that indirection can change which check runs, its execution mode, filtering/suppression, or failure mapping, re-pin the externally-observable contract — not the intermediary's internals: prove the gate still runs in the mode the caller actually uses (not a filtered/quick/subset mode that silently skips it) and that a violation still propagates to a non-zero/blocking outcome. Put the negative test at the narrowest layer that can fail through the re-routed chain; if existing chain-level coverage already proves the A→B→C failure propagation, cite/extend it rather than duplicate. Pure DI, rename, import shim, or extraction with no semantic branch needs no new negative test.
Conditional skips (missing-optional-dependency guards such as module-level importorskip, platform/env markers) combined with per-job test selection can leave an entire test file executed in NO CI job while every pipeline stays green: the job that selects the file lacks the optional dependency (the skip fires for the whole module), and the job that has the dependency does not select the file. When a suite mixes conditional skips with job-scoped test selection, the job that owns those tests must carry an executed-count guard — the per-file invariant and the floor fallback live in references/ci-fixtures-and-flake-control.md. Any change to job-level selection re-verifies which files each job actually executes (run with skip reporting and read the executed/skipped counts per file). "The tests exist and CI is green" is not evidence they ran anywhere.
A new, ported, or mirrored enforcement mechanism (pre-edit hook, permission guard, write-blocking plugin, validator) is not verified by loading, parsing, or config inspection — those prove installation, not enforcement. Require a behavioral matrix before a completion claim — blocked case per deny-condition, allowed/no-collateral cases, the fail-open/swallowed-exception bypass set, and the canonicalization/symlink/worktree edge cases; the matrix cells, the fail-open bypass set, the safe-unavailable-gap disposition for a case that cannot be exercised safely, and the port/mirror parity procedure live in references/verify-enforcement-mechanisms.md. Run the matrix only against scratch/synthetic targets (a throwaway checkout/worktree, fixture repo, or dry-run mode) — never a live workspace, real user data, or live credentials. Parity claimed from code reading alone is hypothesis-grade, not evidence. When a test is ported/mirrored to a sibling stack, input-fixture fidelity is part of that parity — see references/test-code-authoring-patterns.md (跨栈移植：移植对抗输入本身).
A "skip CI" / "no runner" instruction does not by itself lower verification rigor, only ceremony. When the blocking CI gate is skipped or unavailable, substitute a same-risk independent check before treating the change as verified — for a tiny/doc/test-only change a local command or diff --check is enough; for a change that can break its own gate (it edits the test/tripwire/CI config it is guarded by), an adversarial review/challenge of the diff is what catches the self-break CI would have. Note where a local run is not equivalent to CI (secrets, OS matrix, merge-result pipeline) rather than treating it as full proof. Separately, a project-enforced merge gate (pipeline-must-pass, required review) is not waived by a "skip CI" instruction for convenience: require green status, or an explicit authorized break-glass/override with recorded reason + residual risk — surface the conflict and stop rather than silently bypassing.
Browser/E2E smoke must assert visible outcomes, not just click controls. For frontend API pages, verify loading, success, failure, disabled/retry behavior, and absence of dangerous actions where relevant. Capture console errors and failed network requests when tools support it.
UI tests and screenshots must prove design quality layers, not only DOM existence. Assert or visually inspect aesthetic hierarchy/density, interaction path, behavioral recovery states, and psychology-critical cues such as disabled reasons, progress certainty, retry safety, confirmation consequences, and return context.
For every runtime-visible UI/UX slice, load the canonical sequence in ../product-ui-ux-design/references/delivery-contract.md and references/client-runtime-test-matrices.md §UI/UX Delivery Contract before Phase 0 and after producer/client execution. Testing owns layer selection and sufficiency, binds and cites the complete design/test/producer/client record and candidate-binding sets, confirms every affected client wrote its canonical pre-edit client_entry and complete client-record member naming the producer version it exercised, fails closed on a missing/incomplete/mismatched/stale/changed-after-run/unexercised member, and never issues the holistic design verdict.
Authentication and account surfaces need an explicit scenario matrix before they can be called complete. Cover identity-input validation across relevant entries, available sign-in methods, registration, account recovery or password reset/change, logout/account switching, sensitive storage/log cleanup, permission or host-authorization denial, and UI/UX acceptance for error copy, disabled reasons, keyboard/safe-area/touch behavior, and visual evidence. If a capability such as recovery, host authorization, real message delivery, or live account verification is absent or external, record it as product gap, blocked, or live-only instead of silently excluding it from the test claim.
Test cases come before implementation and broad execution for behavior-changing work. Write a compact test-case register first: scenario, layer, assertion, data/dependency, command, expected current result (fail, pass-existing, blocked, infra-error, or gap), and owner. For bug fixes and user-visible or contract-visible behavior, at least one relevant case must be added or updated and run RED before implementation unless no harness can support it after normal remediation; then record the evidence gap and strongest alternate check. The same RED-first discipline applies to defect records: a reported defect (issue, QA finding) carries the repro command plus the actual failing output, and the fix change references that failing test — a bug "fixed" from its description alone, without a RED reproduction, is unverified.
Do not answer "tests are complete" from command output alone. Map each important scenario to a written case or an explicit blocked, live-only, product gap, infra-error, or not applicable row. (verdict definitions and their mutual exclusivity: references/ci-fixtures-and-flake-control.md).
For report-only QA, baseline comparison, or "testing only" branches with no product-code changes, failing tests can be the intended deliverable.
This exception applies only when a human reviewer, PR owner, or user explicitly states in the current work item, PR description, or current-turn context that the deliverable is test coverage, evidence, or a QA report rather than a product fix; an agent or automated process cannot infer or self-apply this exception from prior-session memory or summarized context.
Do not weaken the test or patch product code just to go green.
For disputed or high-stakes defects, prefer splitting verification and fix into two deliverables: a test-only verification slice first pins the confirm/deny verdict and root-cause attribution, and the fix is a separate change that references it — keeping the verification verdict uncontaminated by fix intent. Its failing regression test may land skip-marked as the trace only as a bounded state, not an escape: the skip carries the reason, an owner, and the linked fix item, and accepting the fix requires un-skipping it into the blocking regression set (or an explicitly owner-signed quarantine lane) — a RED test that stays skipped after its fix merges is the bypass this rule exists to prevent.
The QA report is not complete until pass evidence, red-light evidence, blocked/live-only gaps, and baseline comparison are each present and non-empty or explicitly marked not applicable; red-light evidence and baseline comparison cannot both be not applicable.
Live or production behavior is environment evidence, not a correctness oracle: when live behavior contradicts automated or documented expectations, record the discrepancy as a live-only gap with owner; do not resolve it by trusting either side.
A QA report with an open live-contradiction gap is blocked until the discrepancy is escalated and an owner assigns a resolution path.
Generated starter tests are not regression evidence: replace scaffold placeholders in the same delivery slice with assertions for the actual app shell, route, state, or user-visible contract.
Verification warnings are not automatically follow-up work: classify build/bundle/lint/flaky/deprecation/security/perf warnings from a required gate before reporting success — fix now when caused by the current slice or cheaply local; defer only with reason, residual risk, owner, and follow-up artifact.
Do not open, merge, or describe an MR as ready for a contract-visible change until the test matrix is written and executed, or each unavailable layer is explicitly marked unavailable with reason and residual risk. The matrix must include the relevant unit, API/contract, integration, and browser/device/E2E layers; missing layers are release risk, not afterthought.
A multi-stack development-standard family is incomplete without a testing standard. The testing standard must define test deliverables, layer policy, harness expectations, CI gates, high-risk coverage, evidence format, and stack handoff rules; stack docs may specialize commands but must not redefine the layer policy.
Do not mark a browser/device/E2E layer unavailable just because discovery returns nothing. First run the normal remediation path: launch the emulator/browser/server/container, wait for readiness, restart the client daemon if appropriate, run the repo setup script, and re-run discovery. Only after that fails may the layer be reported unavailable, with command evidence, residual risk, and next unblock action.
If a browser/device/E2E or host-smoke layer is classified as blocking, unavailable means the delivery is not complete. Use pre-runtime-test-ready only when code, lower-layer tests, and build checks are done and a named human/device owner must finish the runtime gate; a handoff-only label, not merge-ready/release-ready. Otherwise use blocked. Do not describe such work as done, fixed, merge-ready, or release-ready.
platform-service-connectivity
protobuf-http-contract-signals.md
http-response-envelope-contract.md
rpc-framework-recipe.md
For external/gateway/CORS/log-id/caller-identity exposure triggers, missing, stale, inaccessible, mismatched, or inconclusive contract owner evidence, descriptor proof, owner-suite evidence, boundary-exposure proof, or required sub-check evidence is an explicit open coverage gap, not pass. Detailed platform contract / protobuf / RPC test obligations live in references/integration-contract-testing.md.
Choose the lowest sufficient test layer.
Unit: pure logic, validation, mapping, config, retry decision, state transition, key builder, error translation.
Component/API/contract: handler, generated client/server contract, serialization, auth envelope, route mapping, public API compatibility.
Integration: DB, Redis, MQ, filesystem/object storage, service client, transaction, migration, real parser/generator, dependency adapter.
E2E/real flow: critical user journey, cross-service workflow, browser-visible behavior, release smoke, or runtime integration that lower layers cannot prove.
Scenario matrix: select a small set of representative happy, negative, edge, recovery, permission, and regression scenarios; map each scenario to the cheapest layer that proves it, then add one real-flow check for the highest-risk cross-boundary journey.
A property about what is NOT emitted can only be proven where the emission happens. When the contract says a field/section/header must be absent — not merely empty — under some condition, put absence assertions at the layer that produces the actual wire/rendered artifact — the serialized payload, the response body, the rendered output — and assert key/element absence, not get(key, default) == default, which cannot distinguish the two (the in-process unset-vs-explicit-zero mechanism: references/integration-contract-testing.md). State the split explicitly when the lower layer structurally cannot carry the property, so the reviewer does not read the missing unit test as a coverage gap.
High-risk resilience matrix: for each triggered class, choose the lowest layer that can prove the invariant, then add one release drill or real-flow smoke for the most expensive failure. Examples: idempotency unit/contract plus duplicate callback integration; auth timeout unit plus cross-tenant API integration; AI fallback eval/replay plus visible refusal E2E; client double-submit component test plus server idempotency integration.
High-risk backend tests should include missing tenant/actor/subject/resource-scope rejection, durable idempotency beyond cache TTL, mutation-plus-audit/outbox atomicity or repair visibility, stale/pending worker status, and operator/request/trace evidence for admin repair paths when those risks exist.
For client API-backed surfaces, mini-program/mobile/device runtime smoke, runtime-client mechanisms (route guards, permission trees, request interceptors, generated clients, upload wrappers, long-task polling, safe-area/keyboard/orientation handling, foreground/background restore, native bridges, app-hosted H5), terminal/CLI/TUI runtime tests, and streaming/async-finality changes (model streams, queued jobs, MQ consumers, scheduled prompt tasks, cron tasks, persisted tool-output artifacts, long-running exports, long-lived connections), load references/client-runtime-test-matrices.md before assigning layers — and again before declaring any runtime/device/browser evidence unavailable, not only at layer assignment. Non-negotiable anchors kept in view here: the default three-boundary client split (unit/component + API client/contract + browser/device smoke) applies unless the repository has a stronger convention; runtime-dependent smoke is blocking when lower layers cannot prove the changed behavior, and a missing runner after normal remediation stops at pre-runtime-test-ready or blocked with owner, commands attempted, residual risk, and next unblock action — never converts into a code correctness claim; developer-tool compile/preview is structural evidence only; dangerous or irreversible operations require operator/confirmation/audit/duplicate-submit/final-status assertions before the UI or API is called ready. Build scenario matrices from the reference's reusable dimensions (host/container, identity/permission, data/state, async/finality, visual/interaction, high-consequence); do not paste product-specific matrices into this skill — product-specific lists belong in the project checklist or the owning product/domain skill.
Define data and dependency strategy.
Use small fixtures named by scenario. For repeated/complex fixture construction, pick §4 (factory/builder) from the decision table in references/test-code-authoring-patterns.md.
Use fakes for domain behavior and integration tests for external contracts.
Keep real credentials, live services, long sleeps, and network-only dependencies out of default fast tests.
Tests that hit live DB, Redis, MQ, RPC, model service, object storage, or real external APIs must be marked or commanded as integration/e2e/drill tests. They are release evidence, not the default fast gate, and they must not replace lower-layer assertions.
Use temp directories and cleanup hooks for file output. A temporary smoke/instrumentation test file created only as delivery evidence is removed (or promoted to a maintained test) before the MR/PR (or equivalent review gate) is review-ready — the recorded run output/transcript is the evidence, not the lingering file. Prefer a temp dir or a dry-run/--no-write flag over mutating a real tracked repo file in place. If a test or verification MUST seed or mutate a tracked file, undo only the exact thing you added (delete that line/file) and re-check git status before continuing. In a non-disposable working tree (one with other uncommitted work or that you are not about to delete wholesale), never clean up with a blanket git checkout . / git checkout -- <file> / git reset --hard / rm unless git status first proves your seeded mutation is the only change present — otherwise it also discards unrelated uncommitted edits or deletes committed files. (A throwaway worktree or fresh clone you will remove entirely is exempt.)
Define assertions and evidence.
Assert final user/caller-visible result. Author code per that table — §8: parameterize same-logic inputs instead of copy-pasting — and its closeout checklist walk is required before review.
Assert persisted state, emitted event, dependency call, trace/log id, or browser-visible state where relevant.
For E2E, collect console errors, failed network requests, screenshots/video/traces when useful, but do not rely on artifacts without assertions. Give smoke/e2e harnesses prod-grade rigor: tolerate benign address/format variation (normalize or match structurally instead of pinning a brittle literal) while still asserting the real outcome, and validate the proof artifact itself — reject malformed, empty, duplicate, or stale evidence — so a green cannot be a false positive resting on non-fresh or corrupt evidence.
Place in CI intentionally.
Fast gate: format, compile/typecheck, lint/static checks, focused unit tests, deterministic codegen clean check.
Integration gate: runs only when stable infra or containers exist.
Scenario gate: small acceptance/risk matrix mapped to unit, contract, integration, component, or E2E commands with clear owners for failures.
E2E/release gate: critical smoke and regression flows, isolated environment, explicit owners for failures.
references/fitness-functions.md
For the killing-mutation walk's guarded backup recipe, mutation blast-radius discipline, destructive-artifact probe encoding, and round-trip nuance, read references/run-killing-mutation-walk.md.
For closed-contract oracle design — denylist failure shapes, oracle-dimension explanations, and precision near-miss rows — read references/design-closed-contract-oracles.md.
For enforcement-mechanism behavioral matrices, the fail-open bypass set, and the port/mirror parity procedure, read references/verify-enforcement-mechanisms.md.
