메뉴
Testing pyramid, frameworks, mocking. Load first; then the fragment for the active gate (live smoke, silent failure, bats, legacy triage).
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Schema and migration semantics for /dr-doctor — thin one-liner contract, 6-pass migration, data-loss safety, conflict resolution. Loaded by self-heal.
Core Datarim rules. Load this entry first, then only the fragment needed for paths, storage, numbering, backlog, routing, or archive behavior.
Post-QA hardening — detects task type (code, docs, research, legal, content, infra) and applies the matching verification checklist before archiving.
Preserve Datarim task continuity while orchestrated Claude Code or Codex sessions compact or clear context at deterministic pressure thresholds.
Immutability contract for all pipeline stages: artefact freeze, V-AC parity, non-code parity, anti-tautological rule, and return-to-source transition.
Init-task artefact: verbatim operator brief + append-log, mandatory read by every pipeline command. Source of truth for operator intent.
| name | testing |
| description | Testing pyramid, frameworks, mocking. Load first; then the fragment for the active gate (live smoke, silent failure, bats, legacy triage). |
| current_aal | 1 |
| target_aal | 2 |
Always load this entry first. Detailed gates live in supporting fragments to keep idle context cost low.
When mocking a DB driver (bi.execute = mock.fn(), pool.query = mock.fn(), etc.) in a unit test, the test captures bind parameters before the driver applies its own serialization. Most drivers transform non-scalar bind values on the wire — and that transformation is part of the production write path your test claims to cover. A naive typeof p === 'string' assertion on captured params silently passes for an array bypass that the real driver would JSON-serialize into the column.
Rule. Any unit test that captures DB-driver mock params and then asserts column-shape invariants MUST first run the captured params through a simulateDriverBind(p) helper that reproduces the driver's serialization:
JSON.stringify (default for several SQL connectors when binding to a scalar placeholder — verify per driver).JSON.stringify (when the target column is text; some drivers also auto-serialize for JSON columns).The simulation is driver-specific — declare the helper in the spec file and cite the driver doc that justifies the serialization rules. A simulator for connector X does not work for a test of connector Y; copy-pasting the wrong simulator silently accepts buggy binds.
Why this matters. Application-level sanitization (e.g. a sanitizeValue that maps arrays → scalars) lives upstream of the bind boundary. A regression that lets an array slip past that sanitizer would be invisible to a mock-based assert but would write a bracketed JSON literal into the column on production. The simulator closes the gap a pure-mock spec cannot.
When to apply. Mandatory for every unit test of a DB writer (UPDATE, INSERT, batch upserts) where the column receiving the bind has a documented format contract (scalar, comma-joined list, JSON, etc.). Skip only when the spec is exercising the driver's typeCast / JSON-column round-trip directly — in that case the driver IS the assertion.
A broad unique-violation family such as SQLSTATE 23505 or ORM code P2002
does not identify the domain outcome by itself. Map such an error to a
domain result only when its runtime metadata identifies either the exact
named constraint intended by that mapping or an exact normalized column
signature with set equality against the declared unique columns.
An unknown explicit constraint name wins over column metadata and fails loud; do not fall back to a coincidentally matching column list. When the explicit name is absent, missing columns, partial overlap, subsets, supersets, and every unrelated unique violation remain infrastructure errors rather than domain results.
The required evidence is:
Mutation check: broaden the classifier to the error family alone.
The unrelated unique violation test must then go red. If it stays green, the
boundary is not proved. Missing required evidence is a fail-hard QA/compliance result, never
PASS_WITH_NOTES.
When code under test executes through a framework-internal pass-through — raw runtime hooks where the web framework hands the underlying runtime request/response objects to user code, bypassing the framework's own instrumentation seams — the coverage instrumenter may underreport line/branch execution even though tests pass and the code paths run. Symptoms: a controller/handler whose every behavioural test passes yet shows single-digit coverage, threshold regressions appearing immediately after introducing raw-pass code, branch-vs-main coverage delta with no behavioural delta.
Detection (pre-flight). Before committing the production code, write a placeholder handler with the same raw-pass pattern, run the full test suite under coverage, and compare reported vs. actual execution. If the discrepancy exceeds a 20pp relative threshold against the same code expressed without raw-pass, treat the gap as instrumenter-blind, not test-suite-incomplete.
Remediation hierarchy. Prefer architectural fixes over instrumentation papering:
/* coverage-tool ignore next */) only at the call sites where raw pass-through is unavoidable — typically a single handler line that hands req.raw / res.raw (or equivalent runtime handles) to a framework-internal callback. Each annotated line carries an inline comment explaining why the instrumenter cannot trace it. Do not blanket-ignore whole methods or files.Why this order. Refactor-lift fixes the underlying architecture and yields real coverage; switching instrumenter is a measurement change with no architectural value but preserves test-suite shape; ignore comments are defence-in-depth for irreducible cases. Reaching for the ignore directive first leaves the architectural smell (error logic interleaved with raw-pass code) in place and creates a maintenance debt — the next reader sees coverage green and assumes the raw-pass path is tested when it is merely excluded.
Document the decision. Whichever level of the hierarchy is chosen, record the rationale in the test file or module preamble: which raw-pass call site, which instrumenter behaviour, which remediation level, and (for ignore comments) what the test surface actually exercises. Prevents future contributors from re-litigating the trade-off blind.
A related variant of the same blind spot surfaces when the test runner's include glob is extended to a new directory (e.g. one-off seed / migration / admin scripts living outside the runtime source tree) without a mirroring update to the coverage tool's exclude glob. The test runner correctly executes the spec files (correctness verification works); the coverage tool then measures the source files those specs imply, but cannot instrument the top-level execution path of a script-style module — the file reports zero covered lines even though internal functions are tested. Symptom: a global lines-coverage threshold that passed yesterday fails today by a small fraction (e.g. 79.66% vs 80% gate) immediately after the include glob change; per-file coverage on runtime modules is unchanged.
Rule. Whenever the test runner's include configuration is extended with a path glob outside the runtime source tree, audit the coverage tool's exclude configuration in the same change. Pick one of:
exclude list. Spec files keep running (correctness preserved); only the coverage stat is excluded. Document with one line citing the instrumenter blind spot.Discover this gate locally by running the project's coverage command after the include change — never let the coverage threshold fail at the next archive gate as the first signal.
When QA / Compliance reports cite per-spec test counts (e.g. "added 28 tests" or "11 unit tests in <spec>"), derive each count via a mechanical extractor of the test-runner's case-declaration syntax — never operator memory. The contract is one line: report = output of <extractor> <spec-file>, recorded verbatim in the audit doc. The extractor is a per-language regex whose form depends on the test framework family in use; the rule itself is framework-neutral.
Illustrative extractors (replace with whatever matches the project's test framework):
# nosec-extract
# JS/TS Jest/Mocha-style declaration syntax
grep -cE '^[[:space:]]*(it|test)\(' <spec>
# Python pytest function-style declaration
grep -cE '^def test_' <spec>
# Go testing package convention
grep -cE '^func Test' <spec>
If the audit cites a count that does not match the extractor output for the same revision, treat that as a finding (drift between operator memory and source-of-truth). Source: prior incident — a per-spec count off-by-one in a QA report was caught only by independent re-execution at Compliance.
Commit messages are part of the audit trail too. When a commit-message body cites added test counts (e.g. Tests: N new spec cases or Full suite: M passed (was K)), the numbers MUST come from the same mechanical extractor, not operator memory. Commit messages persist in git history and become the durable record once the branch is pushed; rewriting them after push is destructive (force-push) and after merge is impossible. The recommended canonical form for commit-message test deltas is tests: +N (baseline→total), both numbers produced by running the extractor against the pre-commit and post-commit revisions. Source: prior incident — a commit body cited «13 new spec cases (sub-totals 7+3+6)» where the sub-totals themselves summed to 16, and «Full suite: N passed (was K)» where K was off by 3; both arithmetic mistakes surfaced only at the next pipeline gate. Two-second arithmetic checks belong in the extractor pipeline, not in the operator's head.
When a Definition-of-Done acceptance criterion is a numerical threshold computed by a verdict script over event records emitted by a producer (a daemon, soak harness, ingest pipeline, audit emitter), validate both halves in the same pre-archive gate:
Validating only the consumer half against synthetic events is a recurring trap: the verdict script passes, the producer ships, and weeks later the verdict gate runs against real producer output and exits with «no data in window» because the schema the synthetic test used does not match what the producer actually emits in production conditions.
Rule. A verdict-gate acceptance criterion is incomplete until the archive doc cites one record from the producer's real output stream that the verdict script would consume successfully. A synthetic fixture is not a substitute.
When to apply. Any task that ships a verdict script + acceptance criterion in the same package, where the verdict script is intended to run later against a long-running producer. Skip when the producer is exercised inline in the test (verdict script is unit-tested over the producer's actual output in the same run).
An integration test that spawns a real service process (daemon, server, worker) and connects to it over a socket has a timing surface that a whole-suite run can mask. The canonical stability-verification command is the scoped, repeated run, not the whole-workspace pass:
Common root cause — listen-vs-accept race. A spawned server often prints a "listening on …" / ready line before its accept loop is actually polling. A client that does one cold single-shot connect immediately after reading that line can get an instant connection-refused. Widening timeouts does not fix this (the failure is instant, not a timeout expiry). The structural fix is either a true readiness contract (signal ready only after the accept loop is live) or a bounded client-side connect-retry on connection errors — the latter also hardens the real CLI/client UX against a just-started service.
When to apply. Any task that stabilises or adds a test which spawns a service process and connects to it. Skip for pure in-process unit tests with no spawned subprocess.
A mutation-gate test — one that proves a guard fires by breaking the guard and asserting the observable outcome changes — is only valid when the guard under test is the sole producer of that outcome. When a background reaper, watchdog, supervisor, or cleanup timer runs concurrently and can produce the same observable effect (killing a stuck process, reclaiming a lease, closing an idle connection, evicting a stale entry), the mutation is masked: breaking the guard changes nothing observable, because the ambient reaper still produces the effect. The gate passes green on a guard that no longer works.
Rule. Any timing-sensitive mutation-gate test MUST isolate the guard under test from every ambient reaper/watchdog that can independently produce the asserted effect. Isolate by one of:
Why this matters. A masked mutation is a false green: the mutation test reports the guard is covered and firing, while in production the guard is dead and only the slower ambient reaper is holding the invariant — until a load or timing shift outpaces the reaper and the unguarded failure ships. Verify attribution, not just outcome.
When to apply. Any mutation-checked or timing-sensitive test whose asserted effect (termination, cleanup, reclamation, eviction, timeout) can also be produced by a background reaper/watchdog/supervisor running in the same environment. Skip when the guard's effect is uniquely attributable and no concurrent process can reproduce it.
Source: prior incident — a shutdown-guard mutation gate passed because a background reaper terminated the process on its own timer regardless of whether the guard fired; the dead guard was invisible until the reaper was disabled and the guard probed in isolation.
An integration test that needs a live broker, database, or external service is legitimate to skip when that dependency is absent — but a bare skip (no probe, no message) is indistinguishable from a bug swallowed behind an unconditional skip. A reviewer scanning a green suite with skips has to hand-audit every guard to tell the two apart, which is slow and erodes trust in the suite.
Rule. Any test that needs a live broker/service MUST probe for it in setup() (or the framework's equivalent pre-test hook) and skip with an explicit message naming the missing dependency when the probe fails — never a bare skip, never a fail:
redis-cli -h "$HOST" ping (expect PONG).pg_isready -h "$HOST" -p "$PORT".curl -fsS "$HEALTH_URL" (or the project's health-check convention).setup() {
if ! redis-cli -h "${REDIS_HOST:-127.0.0.1}" ping >/dev/null 2>&1; then
skip "no live Redis at ${REDIS_HOST:-127.0.0.1} — start the broker to run this test"
fi
}
@test "queue consumer processes a job end-to-end" {
# exercises the real broker; only reached when setup() probed it live
...
}
Why this matters. An explicit probed skip is self-evidencing — the message names the exact dependency the reviewer would need to stand up to turn the skip green, so the skip reads as intentional environment-gating on sight. A bare or unconditional skip carries no such evidence; it looks identical whether the author deliberately gated on infrastructure or accidentally short-circuited a broken test, forcing the reviewer to read the guard logic line by line to tell the two apart.
When to apply. Any test whose assertions require a live broker, database, or external service that may not be running in the current environment (local dev, CI runner, sandboxed agent host). Skip this pattern when the dependency is always available in every environment the suite runs in (e.g. an in-process fake or an embedded engine started by the test harness itself) — in that case there is nothing to probe and a missing-dependency skip would never legitimately fire.
Existence assertions ("element renders any text", "counter is non-empty") pass even when the system under test is broken — they only catch the «nothing rendered at all» case. Prefer self-validating assertions that poll the flipped target state after the triggering interaction. Self-validation catches three failure modes in one shape:
Pattern (CDP-driven browser test runners like Playwright / Cypress / WebdriverIO):
await control.click()
await expect.poll(() => readActualState()).toBe(targetState)
readActualState returns the flipped property of the control (a checkbox's checked-ness, a toggle's aria-pressed, a select's value), not "is text non-empty". targetState is the known opposite of the pre-click state. The poll budget is the same number the perf-budget AC declares (commonly 500-1000 ms for a single DOM commit).
Pitfall — native input vs ARIA attributes. A native <input type="checkbox"> does not set aria-checked (the attribute is meaningful only on role="checkbox" custom controls); query the native checked property instead. Component-library wrappers (MUI / Chakra / Mantine / etc.) vary — confirm by inspecting the rendered DOM once before authoring the spec, not after the spec flakes in CI.
Source. A perf-budget assertion was first authored as «counter renders any text» and passed even when the toggle was visibly broken; rewriting it as «poll isChecked() against the flipped target» surfaced the regression class deterministically.
When to apply. Any browser-driven E2E that asserts «interaction X commits state Y within budget Z». Skip for purely visual assertions (computed style audits, screenshot regression) — those are their own pattern.
When a diff introduces or modifies a defensive write-gate guarding a shared target (column, field, document property, in-memory key — any sink that more than one code path can write to), the gate is not safe in isolation. Enumerate every other writer in the surrounding scope that can land a value at the same key, and add one regression test per distinct writer-pair semantic.
Rule. For each writer-pair (gate writer plus one other writer in scope), the regression test must seed the case where writer A produces a non-trivial value X and the gate writer B fires on the same key in the same pass. The assertion checks (a) the documented winner is written, (b) no silent clobber of the loser, (c) no false-fire on the gate's diagnostic emissions. A defensive gate accepted on single-writer coverage alone will silently overwrite the parallel writer's fresh value the first time the two paths converge in production.
Why this matters. A silent operator-data-loss regression of this exact shape — gate writer unconditionally assigns into a shared key, the parallel writer's value lost — is invisible to any test that only exercises the gate's intended path. Single-writer coverage proves the gate fires; it does not prove the gate respects concurrent writers. The class is structurally invisible to per-method unit tests; only writer-pair tests catch it.
When to apply. Any change to a write-gate, sanitizer, repair pass, or boundary-defence layer where two or more code paths in the same scope can write to the target key. Skip only when the gate's target key is provably write-once (e.g. immutable after first assignment, enforced by language or schema). Document the disjointness claim in a code comment so future readers can re-verify when the surrounding code shifts.
Where it lives in the diff. Plan-time: enumerate writers in the implementation note before code. Code-time: add one spec block per writer-pair. Review-time: a reviewer scanning the diff should be able to read the writer-pair list and match each pair to a spec block.
When writing a test for a defensive gate, identify every upstream layer (sanitizer, transform, boundary-defence pass, normaliser, validator) that would have already handled the defended-against state before the gate ran in production. Stub those layers to passthrough, not to reproduce the problematic state the gate is defending against.
Rule. A defensive-gate test must exercise the gate against a realistic production-failure input — the shape that actually reaches the gate when an upstream layer is failing, absent, or out of date. If the test setup seeds the problematic state upstream of the gate, the test passes through layers that would have already normalised the input in production, and the gate is never genuinely exercised against the failure mode it is designed to catch.
Why this matters. A test that seeds a residual literal at the top of an extraction chain and lets the production sanitizer strip it before the gate runs will pass without ever testing the gate. The gate could be removed entirely and the test would still pass. The failure mode the gate is designed to catch — residue surviving the sanitizer chain and arriving at the write boundary — only appears when the sanitizer-chain stub is passthrough, i.e. when the test reproduces the production-stuck shape, not the dev-happy-path shape.
When to apply. Any unit test of a defensive gate (repair pass, write-boundary defence, normalisation guard) where one or more upstream layers in the production pipeline would have already handled the defended-against state. Stub each upstream layer to passthrough and seed the problematic value directly at the gate's input boundary. Skip when the gate has no upstream layers (the gate is the first stage in the pipeline); in that case the dev-happy-path input shape coincides with the prod-stuck shape.
How to identify the right stub layer. Walk the call chain from request entry to the gate. List every transform that would touch the gate's input slot. Stub each transform to identity. Run the test once with the stubs in place and verify the gate's input matches the prod-stuck observation that motivated the gate's existence in the first place.
When a skill, agent, command, or in-code docstring documents the behaviour of a parser, library, or runtime quirk — especially edge cases or fail-soft caveats — runtime-probe every claim before commit. A one-line CLI invocation that demonstrates each documented case takes about 30 seconds; the cost of detect-correct at iteration N during a multi-iter verification cycle is hours. The class extends beyond any single library — operator-facing precision matters even for fail-soft caveats, because an inaccurate caveat produces wrong-shaped operator mitigation.
Rule. For every behavioural claim in a docstring or skill paragraph that describes how a third-party parser, library, or runtime behaves on edge input (<!-- example: empty marker is parsed as comment-->`), demonstrate the claim with a runnable probe. Capture two cases minimum: one that confirms the claim, one that contrasts an adjacent shape the reader might confuse with the documented case. Document the contrast inline so the docstring tells the reader where the boundary actually lies.
Why this matters. A docstring whose claim was never runtime-verified at write-time will surface as a finding at the next peer-review pass — sometimes many iterations later, with the inaccurate guidance already shipped to consumers in the meantime. The remediation also costs more: by the time a reviewer flags it, the surrounding context has shifted and the author has lost the mental model of the original probe.
When to apply. Any new or modified docstring, skill paragraph, or command instruction whose body asserts how an external parser/library/runtime behaves. Skip only when the claim is purely procedural ("call function X with arguments Y") with no behavioural assertion about a third-party surface.
For test-first discipline (RED-GREEN-REFACTOR cycle, the Iron Law that no production code ships without a failing test first, the rationalization table that pre-answers "I'll test after / it's too simple / TDD is dogmatic"), load tdd-discipline.md. Apply when implementing any feature or bugfix in a context that warrants TDD — see the entry skill's Mocking Rules and § Live Smoke-Test Gates for what counts as a real test.
Load only the fragment needed for the current sub-problem:
tdd-discipline.md
Use whenever you would otherwise write production code without a failing test first. Mandates the RED-GREEN-REFACTOR cycle, captures the common rationalizations and the canonical responses, and lists the red-flag phrases that mean STOP and start over.
live-smoke-gates.md
Use for raw-SQL / cross-datasource Live Smoke-Test Gate, cross-container Docker smoke, user-switch deployment gates, N=1 smoke validation before bulk ingest/transform, recorded-fixture tests for thin HTTP wrapper clients, and Gate 9 child-failure attribution for status-transforming shell harnesses. Trigger when the change touches $queryRaw, multi-datasource code, Docker orchestration, container health, runtime user/permissions, any bulk run that depends on entity resolution / record linkage / normalization, a new/changed wrapper client around an external/internal HTTP service, or a shell layer that turns a child result into QA/compliance evidence.
silent-failure-detection.md
Use for wrappers around CLIs/subprocesses that exit 0 on error and write error sentences to stdout (LLM CLIs, cloud tools). Mandates structured-output parsing, raise-inside-wrapper, and testing both exit-code scenarios.
bats-and-spec-lint.md
Use for shell-script testing with bats-core (isolation, fixtures, alignment tests, SUCCESS markers, sanity guards) and for spec-lint regex assertions over markdown prose contracts.
triaging-legacy-failures.md
Use when inheriting a test suite with pre-existing failures. Three-bucket triage: stale-delete, fixable-patch, rephrase-the-content.
${DATARIM_RUNTIME:-$HOME/.claude}/templates/docker-smoke-checklist.md — 5-step reusable checklist for cross-container smoke (Compose validity → container health → endpoint smoke → end-to-end action with post-conditions → rollback). Reference this when applying the Live Docker Smoke gate.live-smoke-gates.md.live-smoke-gates.md § Gate 9; prove positive control, negative control, and exact named-assertion attribution.live-smoke-gates.md § Gate 7 (Agentic Entrypoint Wiring + Live-Run): prove the real entrypoint reaches the declared function AND run it once live against the real tool before any "the agent does X via " wish can be marked met.silent-failure-detection.md.bats-and-spec-lint.md.triaging-legacy-failures.md.Testing is in the hot path for every QA / /dr-do / /dr-qa flow. A 426-line monolithic file forced every agent to load the full content even when only one gate was relevant. Split into entry + 4 supporting fragments reduces idle context cost while preserving the full contract.