testing-strategy

name

testing-strategy

description

怎么测 / 测试方案 / 测试覆盖 / 先写测试用例（可执行测试代码/覆盖） / test cases first for test code / 写测试 / 写测试代码 / 补测试 / 补测试用例（未指向文档/Bitable/source-to-TC 时按可执行测试覆盖处理） / mock / 单元测试 / 集成测试 / E2E / 回归 / CI gate → choose test layers, scenarios, fixtures, mocks, regression coverage, smoke, verification evidence, and runner-config policy. Skip structured test-case documents, Feishu/Bitable TC sync, or source-to-TC generation → test-artifact-management. Skip 具体的 runner 调用（pytest / go test 命令与参数怎么写）→ 该技术栈 dev 技能；覆盖阈值与 CI gate 策略仍归本技能。

Testing Strategy

Use this skill when the task is about test scope, test design, verification evidence, CI gates, or deciding which layer should prove behavior. It is stack-agnostic: apply it across Go services, Python services, React web, mobile apps, mini-programs, terminal/CLI/TUI surfaces, APIs, workflows, data/inference features, and product releases.

Skill Routing

• Use this skill for test strategy, test pyramid decisions, unit/integration/e2e/scenario boundaries, fixtures, mocks/fakes, regression coverage, and verification evidence.
• Use go-microservice-dev for Go-specific implementation details after the test layer is chosen.
• Use python-service-dev for Python-specific pytest, async test, fixture, fake client, ruff, mypy/pyright, packaging, or migration-test implementation details after the test layer is chosen.
• Use app-cross-platform-dev for Flutter unit/widget/integration tests, Android unit/Compose/instrumented tests, iOS unit/UI tests, mobile device smoke, and app release verification after the test layer is chosen.
• Use miniapp-product-dev for WeChat/Alipay/Douyin/Baidu mini-program build, developer-tool preview, real-device smoke, host capability verification, review submission, and release checks after the test layer is chosen.
• Use web-react-dev for React unit/component/API-client/browser/E2E implementation, browser automation, accessibility checks, and build/deploy smoke after the test layer is chosen.
• Use terminal-cli-dev for command-line, terminal, PTY-backed, ANSI-rendered, keyboard-driven, and full-screen text UI implementation after the test layer is chosen.
• Use llm-inference-integration for LLM/agent/RAG evals, replay sets, prompt/model regression checks, shadow runs, and inference-specific observability after the test layer is chosen.
• Use defect-diagnosis first when a test is failing, flaky, or used to reproduce a bug.
• Use product-rd-workflow first when testing is part of an end-to-end product delivery plan.
• Use product-ui-ux-design for visual hierarchy, interaction, state, accessibility, and design acceptance criteria; use this skill for automated proof and test coverage.
• Use test-artifact-management when the ask is about generating structured test cases from a Feishu requirements doc or codebase and delivering them to Feishu Bitable; use this skill for scenario selection and risk matrix before test-artifact-management writes the cases.
• For ambiguous "写测试用例" / "write test cases" requests, use this skill first when the user is asking to add, modify, run, or verify test code, E2E/browser/device tests, CI coverage, mocks, or regression evidence. Route to test-artifact-management only when the deliverable is a structured TC artifact, Feishu/Bitable sync, or source-to-TC generation workflow.
• For team testing standards, QA norms, CI/harness requirements, or a missing "测试规范" in a product/R&D standards family, use references/rd-testing-standard-template.md as the source-neutral template, then route stack-specific command mechanics to the relevant dev skill.

Non-Functional And Specialized Testing

Use this skill to decide whether a specialized or non-functional test belongs in a gate and what evidence is required; the domain owner still owns implementation semantics and detailed method. Keep the entrypoint as the ownership/pointer surface. The full domain index, per-domain gate criteria, scenario/assertion matrices, and routing handoffs live in references/non-functional-specialized-scenarios.md.

Core Rules

• Test behavior and contracts, not implementation trivia.
• Code analysis may add risk scenarios; it must not define product-delivery completeness. Derive primary coverage from the active requirement/acceptance source with stable IDs.
• Do not write change-detector tests — assertions that snapshot data expected to change (vendor/catalog entries, config-version literals, enumeration counts, hardcoded registry lists) instead of behavior; they add no coverage and break CI on routine data updates. Reviewers convert them to relationship invariants plus a minimal product-guaranteed anchor, never transient values. Keep a literal only when it traces to a documented product/API/security contract or past defect, not implementation state, and the test name states that contract; otherwise, after confirming no behavior coverage is lost, convert it to a relationship invariant. Heuristic, exceptions, and examples: references/test-code-authoring-patterns.md §3.
• Every test should pay for itself: prioritize by business risk, defect history, change frequency, and manual-verification cost.
• Reuse before generating. Before writing any test, grep existing test files for coverage of the same function, endpoint, class, or scenario. If coverage exists, extend or parameterize it instead of creating a new test function or file. Duplicate test structure wastes maintenance and produces misleading coverage numbers.
  • When the change CORRECTS a behavior (a compliance, contract, privacy, or spec-conformance fix), that same grep has a second job: find the assertions that PIN THE OLD, NOW-WRONG BEHAVIOR — and sweep every layer that asserts it, not only the unit tests you are already editing. Rewrite each pinned assertion to the corrected contract in the same change (never delete it to go green; the corrected form is what stops the old behavior from returning). Why the old behavior is always pinned somewhere, and how to enumerate the layers to sweep: references/test-code-authoring-patterns.md §10.
• Keep default tests fast, deterministic, and offline. Select local evidence by touched surface. Leave the matrix to CI only if verified blocking jobs cover it; otherwise run the missing gate or mark incomplete. Report commands run; reuse green only on the same recorded tree (references/ci-fixtures-and-flake-control.md, Local Evidence Selection).
• Do not infer coverage from test file count or names. Classify tests by the command that runs them, marker/tag, required dependencies, CI placement, assertion strength, and whether failures block delivery. An uncovered line is first a delete candidate, then a test candidate (references/test-code-authoring-patterns.md §6).
• Classify test evidence quality before using it in a completion claim: strong tests are deterministic and assertion-based; medium tests assert behavior but depend on fixed DB/cache/network state; weak tests only log/print, sleep, use random/live data, call live model services, or sit behind non-blocking CI. Weak tests can inform risk but cannot prove correctness. Evidence about an agent's own output observes the world, not its self-report (, Verify The World).

Entry Decision: TC Source and Scope

Before any test design or implementation work, resolve two questions: where do the test cases come from, and what is the scope.

Step A — Resolve acceptance and TC sources

1. For behavior-changing product delivery, always resolve the acceptance source first. Record acceptance-source lookup: found <artifact/section> or acceptance-source lookup: absent <locations checked>. When found, preserve the complete point-ID set and source decisions; TCs may refine but not silently narrow it. When absent, functional completeness is blocked: route to product-rd-workflow to establish the acceptance inventory — test-artifact-management may only run after an authoritative inventory exists, because it can generate cases from existing code and would reproduce the omission blind spot; code analysis remains supplemental. A delivery carrying an entry-classified functional-axis exemption claim (documentation-only, pure refactor, or mechanical maintenance per product-rd-workflow; the reviewer confirms or rejects the claim at closeout) is outside this step — cover it as existing-behavior/refactor coverage under step 4.
2. User provided TC URL or file path → fetch and use it directly.
3. .report-config.json exists in the working directory → auto-read base_token + table_id; fetch active TCs from Bitable via lark-cli base +record-list; filter out 废弃 records before passing to the implementation queue. Do not ask the user — auto-read is the default.
4. No TC source found → code analysis may define existing-behavior or refactor coverage. For product delivery, continue from the acceptance source, record the TC gap, and use code-derived scenarios only as supplemental risk coverage.

When TCs are available, pass the full list through When Input Is a Structured TC List and references/structured-tc-input-translation.md before doing anything else.

TC coverage vs. test code coverage — expected gap: TCs cover functional and interaction scope (QA perspective: user journeys, acceptance criteria, error paths, permissions). Test code covers a broader surface: unit-level logic, internal invariants, boundary conditions, data validation, refactoring safety nets. These two scopes do not and need not align 1:1.

• Test code without a TC ID is normal and expected. It runs as part of code quality coverage and does not sync to Bitable. Do not force-add TC IDs to tests that have no corresponding Bitable entry — that creates phantom TC links. When the underlying code is removed or deprecated, these tests should be reviewed and the user prompted to decide whether to delete them.
• A TC entry without automated test code is also valid: manual test, deferred automation, or blocked environment.
• The automated report (gen_report.py) tracks TC-mapped results (tests that register TC IDs via the tc(...) helper — see test-artifact-management/references/tc-marker-conventions.md) plus a separate "未链接 TC 的测试" section for tests without TC links. Broader code coverage is a code quality concern tracked separately (e.g. coverage reports, CI pass/fail).

Step B — Determine scope (if not already bounded)

A current acceptance source or reviewable task artifact may already confirm scope. In particular, a UI/UX Design brief with a stable slice/surface, authoritative consumer inventory, affected owner(s), and criterion IDs is the confirmed bounded scope for its Phase 0/Phase 1 work; consume it instead of asking the user to restate global/file/function scope.

If neither the request nor a current authoritative artifact bounds the work, inspect the current task, repository contract, diff/target, and available acceptance sources first. Ask only when two or more plausible scopes remain and choosing among them would materially change the test plan. Then ask the smallest concrete question, for example:

请确认测试范围：

1. 全局 — 整个仓库 / 当前 feature 所有文件
2. 文件 — 指定文件（请提供路径）
3. 函数 / 接口 — 指定函数或 API endpoint（请提供名称）

Do not infer scope from stale conversation or an unverified guess. A current, resolvable Design brief or accepted scope artifact is evidence, not inference. If the inspection leaves one material scope, proceed and record its source; if ambiguity remains, wait for the user's answer before layer assignment.

When scope is confirmed:

• 全局: run the full scenario matrix from testing-strategy workflow; consult TC list for all active TCs.
• 文件: narrow TC list to TCs whose 模块 or 功能点 maps to that file; run scenario matrix for that file's contracts and business rules.
• 函数 / 接口: narrow to TCs directly covering that function or endpoint; add negative-path and boundary cases if not already in the TC list.

After resolving TC source and scope, proceed to When Input Is a Structured TC List and references/structured-tc-input-translation.md (if TCs available) or Workflow Step 1 (if no TCs).

---

When Input Is a Structured TC List

When the input is a set of structured test cases from test-artifact-management or a Feishu Bitable TC list, use references/structured-tc-input-translation.md as the translation guide, then run the standard workflow from Step 1. The mapping is directional — not every TC becomes one automated test, and one TC may need assertions at multiple layers.

Keep these entrypoint anchors in view while translating: normalize TC status before implementation (full status table in the reference); treat 失败 as a regression candidate requiring RED-before-implementation or a named evidence gap; treat 阻塞 as blocked with owner/gap; exclude 跳过 / 废弃 from the implementation queue without deleting or overwriting Bitable records; treat P0/P1/P2 as business risk signals, not direct layer or CI-gate assignments; keep TC IDs in sidecar/helper metadata rather than test function names; let this skill choose the layer/gate, stack skills implement mechanics, and test-artifact-management own TC content changes and Bitable sync. After pre-check and field mapping, run the standard workflow from Step 1 (discover repository test topology).

Workflow

Before editing tests, CI gates, mocks/fakes, fixtures, test scripts, verification docs, or implementation tied to a testing request, complete enough analysis and planning for the testing change to be reviewable. Scale the plan to risk: a simple low-risk single-test addition can use a short inline plan; multi-layer, user-visible, contract-visible, flaky, release-blocking, bug-regression, branch/MR, unclear-risk, or high-risk work needs explicit scenario/risk matrix, test-case register, task split, expected failing test or evidence gap, verification commands, and stop conditions before edits.

1. Discover the repository's test topology.

   • Read local guidance first: AGENTS.md, CLAUDE.md, README, CONTRIBUTING, local agent skill/instruction directories, Makefile, package.json, pyproject.toml, go.mod, pytest.ini, vitest/playwright config, CI jobs, scripts/run_tests.*, and scripts/dev.*.
   • Identify default fast tests, opt-in integration markers, contract/architecture checks, browser/E2E suites, local container stacks, and release smoke commands.
   • Exclude dependency and generated-test noise from the topology: node_modules, virtualenvs, vendored/third-party trees, generated SDK snapshots, build outputs, and archived worktrees unless the task is explicitly about those artifacts.
   • Prefer documented wrappers over raw runners unless the task is to debug the runner itself.
   • If a frontend or app repository has only build/dev/format scripts and no assertion-based UI tests, do not treat that as adequate coverage for interaction changes. Use build/typecheck as a structural gate, then add focused state/API tests or record the missing test layer and require rendered browser/device evidence.
   • If CI mainly builds images or deploys by branch/environment, classify it as release plumbing. It does not prove product behavior unless assertion-based test jobs run and block delivery.
   • When adding a new client platform scaffold, inspect whether the generator added default sample tests. Keep a sample only if it asserts real product behavior; otherwise replace it with the smallest deterministic smoke that would fail if the current shell, navigation, entrypoint, or visible state were missing.

2. Identify the behavior to prove.

   • User/caller outcome.
   • Contract, state, side effect, emitted event, rendered UI, permission behavior, or operational signal.
   • Risk if wrong.
   • Scenario dimensions when relevant: persona/role, entry path, data state, permission, dependency health, device/browser/platform, locale/timezone, network quality, and recovery path.
   • For API contract changes, first record the contract definition being tested: request shape, response envelope, success/error semantics, compatibility mode, consumers, and rollout assumption. If this is missing, stop and route back to product/architecture before writing broad tests.
   • For protobuf-backed HTTP, JSON/OpenAPI vs protobuf wire-format, and internal RPC/base/generated DTO paths, use the canonical policy/proof references from : , , and . This skill owns assertion coverage, verdict shape, and CI placement; platform-service-connectivity owns the platform contract semantics and proof mechanics.

Reference Loading

• For source provenance, current extraction boundary, and keep/merge/discard decisions, read references/source-evidence-map.md when auditing or re-extracting this skill.
• For discovering repo-specific commands, test directories, wrappers, and CI placement, read references/test-topology-and-commands.md.
• For scenario matrices, user journeys, acceptance coverage, and mapping scenarios to test layers, read references/scenario-testing.md.
• For unit tests, fakes, table tests, and deterministic fast targets, read references/unit-testing.md.
• For test data, factories, fixtures, fuzzing, property tests, and deterministic controls, read references/test-data-and-determinism.md.
• For API, contract, DB, Redis, MQ, generated code, and dependency integration tests, read references/integration-contract-testing.md.
• For client API-backed surface layer splits, mini-program/mobile/device runtime smoke rules, runtime-client mechanism tests (incl. native-bridge and app-hosted H5), terminal/CLI/TUI runtime-test layer choices, reusable scenario dimensions, and streaming/async-finality matrices (model streams, queued jobs, MQ consumers, scheduled prompt tasks, cron tasks, persisted tool-output artifacts, long-running exports, long-lived connections), read references/client-runtime-test-matrices.md (Workflow step 3 routes here).
• For batch jobs, ETL/ELT, streams, scheduled jobs, backfills, replay, and data-quality assertions, read references/data-and-workflow-testing.md.
• For browser, API workflow, real-flow, release smoke, and E2E evidence, read references/e2e-real-flow-testing.md.
• For CI gates, test data, fixtures, flake control, and verification reporting, read references/ci-fixtures-and-flake-control.md.
• For translating structured TC lists from test-artifact-management or Feishu Bitable into test-layer choices, status handling, TC ID metadata, execution-type routing, and handoff rules, read references/structured-tc-input-translation.md.
• For how to write the test code itself — the 8 stack-agnostic authoring patterns (structure, naming, smells, fixtures, behavior-vs-state, coverage, isolation, parameterization) plus the closeout checklist — read references/test-code-authoring-patterns.md (steps 4–5 route here).
• For architecture fitness functions (Ford et al. 2017) — automated tests that verify architectural invariants (no circular deps / DAL-only DB access / latency budget / module-boundary enforcement / breaking-change detection / security invariants), read . Different from product correctness tests; protects structural commitments declared in ADRs from drift.