dev-testing

# Testing Read `tests/README.md` before adding or moving test fixtures. Test-only config belongs under `tests/fixtures/`, not root `config/`. ## Where a new guard belongs A test that records an architectural mistake -- something that must not be repeated, checkable from source -- is a Citadel guard, in `tests/citadel/`, not another file at `tests/` root. It runs in the fast phase, so it costs seconds rather than waiting on an asset build. Two rules the existing guards follow: State the reason in a named `*_RATIONALE` appended to the assertion, so a violation teaches. `test_db_boundary.py` is the model. Where the reason is already stated canonically -- `config/gate.toml [boundary]`, AGENTS.md, a skill -- cite it instead of restating it. Write the adversarial case, not just the conformance case. A guard proved only by inputs it was built for is not proved. `test_workflow_enforcement.py` carries fourteen evasions and four legitimate shapes because enumerating bad spellings let five through; the rule is now a whitelist for that reason. ## Test execution Write code first with TDD and minimal focused feedback. During routine edits run the regression being changed and only necessary file/behavior checks. Broad workspace, contract, Citadel, VM, installed-package and performance runs belong at major phase boundaries, not after every edit or commit. Pending full proof must not block independent implementation. Qualify the complete phase before calling it qualified, and retain the exact-source merge/release gates. The bounded wrapper provides timeout and cleanup without a default machine lease; reserve `--machine-lease` for explicitly isolated qualification work. Finish a focused build/test command before editing its inputs. Cargo may have already compiled a test when a later edit lands, so its green result can cover older bytes. If inputs changed during compilation or execution, that result does not verify the final source; rerun the focused regression after the edit. SDK acceptance launchers use `uv run --frozen --no-sync` against the environment prepared by the gate. Snapshot refresh changes source mtimes; an implicit sync can rebuild the editable SDK and fetch its backend inside the sealed namespace. Dependency preparation belongs to the earlier environment owner. Linux managed-supervisor acceptance hands the private manager's runtime and PID explicitly through the privilege transition. The worker checks its socket peer, native owner and network namespace before restoring the manager environment. Never adopt an ambient user manager outside the sealed worker's namespace: its service can be healthy while localhost is unreachable from the test. Pass that authenticated manager's runtime and bus address explicitly into the service unit too. A user manager's default unit environment can point at a different session; worker-only binding leaves a healthy service unable to verify its own restart policy. Focused Ruff commands must select `--config build_system/pyproject.toml`, as the gate does. Auto-discovery from a root-level test used a different Python version and reversed `tomllib` import grouping, passing locally then failing the canonical source phase. | Command | What | VM? | |---------|------|-----| | `just fast-test` | Incomplete source feedback only | No | | `just focus-test <group>` | One existing owner; `rust` selects changed crates and reverse dependents | Depends on group | | `just install` | Complete local macOS package build and native install for hands-on testing | Yes | | `just test` | Complete local verification; required, passing, before a release | Yes | A release requires a passing `just test` for its source first; hosted release lanes then qualify what they publish. `just fast-test` and `just focus-test <group>` are broader feedback tools for a phase boundary or a failure that needs their scope. During TDD use the smallest native test directly; use complete verification at major phase boundaries. Pick the narrowest lens, in this order: one test file run directly with pytest in the latest `cache/worktrees/<run>` (it reuses that run's built assets; rebuild and re-sign only what changed), then `focus-test <owner>` (`kingslanding` for containers, `assets` to rebuild the guest), then `functional --slow` once on the final head. Ask the user before adding a new owner. The full gate is a construction boundary, not the edit loop. During TDD, reproduce the failure with the smallest focused test, run that test red/green, and batch adjacent parity fixes before paying for the complete gate. Run it only when explicitly requested. Later fixes use focused owners; a gate-policy edit is not permission for another full run. The one full run a change always needs is the passing `just test` a release of it requires; the release command refuses without it. `just test` deliberately accepts committed or uncommitted developer work. It records `HEAD` plus a digest of every tracked and untracked non-ignored source byte, then fails if either changes while the gate runs. Generated output stays under ignored build directories. The local proof therefore covers the exact source state the developer asked to test without forcing commit choreography. Automatic gate benchmark output belongs under ignored `cache/target/tests/benchmarks/`. Historical benchmark publication uses the owning pytest/benchmark command and an explicit review; it is not a Just convenience recipe. ## Release CI invariant Release CI reuses the same checked-in private modules as local `just test`: - `_test-fast` - `_test-static` - `_test-artifacts` - `_test-functional` - `_test-glowup` - `_test-release-contracts` The build scope is selective; the quality scope is not. Binary CI builds only packages and resolves the channel's runtime by recorded digest. Runtime CI builds the runtime for one channel and resolves the current package by recorded digest. The resolved complementary artifacts are staged into the production test harness, not replaced by source-built substitutes. Every activated pairing must pass artifact validation, all VM suites, Winterfell/MCP lifecycle, IronBank, injection, integration, benchmarks, full `capsem-doctor`, exact native install, and update glow-up. A staged runtime whose minimum package is not yet satisfied may run only the self-consistency, integrity, isolation, and boot proof; the following binary lane must run the complete functional and glow-up modules before activation. Both macOS and Linux must install their exact publishable native packages, including their real post-install scripts, before publication. Notarization and the public stable-to-nightly switch/upgrade glow-up remain mandatory end-to-end proof. On Apple Silicon macOS, `just test` owns the pre-publication macOS package boundary through `build_system/packaging/macos/macos_release_glowup.py`: it builds the package with the production assembler, installs that exact file in a disposable headless Tart guest, verifies the receipt, app bundle, complete binary cohort, service and gateway health, then extracts the same package on the physical Mac and boots a real Capsem guest VM from its exact binary/runtime payload to a shell marker. The local package is unsigned; its postinstall ad-hoc signs the installed Mach-O payload with the required entitlements. Local qualification must not load Developer ID material or create a signing keychain. The tagged publication workflow alone signs, notarizes, staples, and installs the final publishable package. Tart macOS guests do not support nested virtualization, so these are two explicit halves of one script rather than a claimed nested proof. `just fast-test` deliberately excludes Tart and therefore cannot be used for release. Rust is pinned to `1.97.1` across the workspace file, workflow steps, host-builder, and bootstrap. Change all pin surfaces together in a deliberate toolchain-bump PR. RustSec and JavaScript bulk advisories are blocking in `just fast-test`, local `just test`, ordinary CI, and both release lanes, as well as the scheduled/manual audit. A new advisory fails the candidate until it is remediated or explicitly reviewed in checked-in scanner policy. Linux proof is host-aware: a cross-built non-host package receives structural validation in qualification and exact native installation in its tagged release job. `CAPSEM_REQUIRE_LINUX_DEB_PROOF=1` must not reject that non-host package before the host package reaches its mandatory KVM proof. The hosted arm64 runner does not expose `/dev/kvm`, so it proves exact package/service operation while the x86_64 runner additionally owns the guest-shell marker. Expensive harnesses need a cheap clean-environment bootstrap proof at the start of `just test`, before Docker/Colima or artifact preparation. The one private `_test-fast` module is also called by `just fast-test`, ordinary CI, and both release lanes. It owns YAML/workflow and source syntax, source contracts, dependency audits, Clippy, Python lint/type checks, and JavaScript/frontend checks; no caller may reproduce a subset inline. Only a green fast gate may build the Linux install-test image. That preflight must use a container-owned `UV_PROJECT_ENVIRONMENT` and prove `python -m pytest` launches before VM, package, or asset work consumes hours. Keep an ordering contract. It is fail-fast infrastructure validation only: the later Docker/systemd install E2E remains mandatory and must still exercise the installed package and post-install behavior. ## Local/CI execution parity Read `references/local-ci-parity.md` before editing any release workflow, gate recipe, or CI job. It holds the Ironbank parity rule (every portable release gate must be owned by `just test`), scanner/tool pinning, Docker platform and prune discipline, common cache contracts, and the source-guard contracts. ## TDD workflow Write tests first: 1. Write failing tests that capture expected behavior 2. Verify they fail for the right reason 3. Write minimal implementation to pass them 4. Refactor Without a failing test first, tests can pass by accident. Cargo fixtures, including tiny probes, use `tests/helpers/bounded.py`: its `env` argument applies explicit probe compiler and target overrides after containment; never serialize inherited secrets. The wrapper owns the finite child timeout and cleanup; routine probes do not acquire a machine lease. Do not add an outer subprocess timeout. ## Functional slice proof matrix Use the matrix below for qualification at each major phase boundary. During implementation run only the focused checks needed for the changed behavior and record outstanding phase proof. A green unit suite or a benchmark is not a substitute for complete phase qualification. | Category | What it proves | Minimum expectation | |----------|----------------|---------------------| | Unit/contract | Pure logic, parser state machines, schema migration, helper APIs | Red/green tests for normal and edge behavior at the smallest useful boundary | | Functional | The feature works through its production-facing API, not just private helpers | Exercise the real module boundary with realistic inputs and assert outputs plus side effects | | Adversarial | The feature preserves security, privacy, and policy invariants when attacked | Malformed, oversized, denied, missing, racing, timeout, permission, and leak-prevention cases | | E2E/VM | The user-visible path works in a real Capsem session | Boot/run a VM or use the black-box CLI/MCP/service path, then inspect externally visible behavior | | Telemetry | Audit data is present, accurate, and queryable | Query `session.db` or logger readers for required rows, fields, decisions, errors, and attribution | | Performance | Hot paths stayed inside the accepted budget | Benchmarks or timing assertions with recorded numbers and regression criteria | If a category is genuinely impossible or deliberately deferred, record it as missing with a reason, owner, and follow-up task. Silent deferral is the bug. "Covered by later E2E" is not enough unless the tracker names the later test and the current milestone is explicitly scoped as internal-only. For policy, MITM, MCP, telemetry, networking, filesystem, process lifecycle, or sandbox-boundary work, the functional slice matrix is mandatory. The tests should prove not only that the happy path succeeds, but also that enforcement happens at the intended boundary: a blocked MCP tool does not dispatch, a blocked return does not leak, a denied URL does not reach the network, a malformed frame does not poison the stream, and telemetry records the truth. ## Ironbank ledger tests Use `/ironbank` for release-critical VM, network, model, MCP, credential broker, package-manager, doctor, benchmark, and security acceptance proof. Ironbank lives in `tests/ironbank/` and is full black-box: tests are written from public contracts, CLI help, docs, generated schemas, hermetic fixtures, route responses, logs, DB rows, and installed package metadata. Do not inspect Rust/product internals to decide expected behavior. Ironbank cannot use: - `skip`, `skipif`, `slow`, optional markers, or public-network dependencies - status-code-only replay - row-exists checks - parser-only assertions - manual OAuth/client runs as release proof One deterministic stimulus must prove the whole ledger path: client result, parsed facts, CEL/security decision, detection/enforcement rows, protocol DB rows, structured logs, status counters, UDS route, HTTP route, and UI-facing JSON. Every emitted DB/log/route field is exact-value asserted, covered by a typed invariant, or explicitly marked not applicable. Unknown fields fail the test until the field ledger is updated. Package-manager tests prove function. Installing `zstd`, for example, means compressing known bytes, decompressing them, and comparing the exact output; not just checking dpkg output. ## Logged-data DB ownership Telemetry and security ledgers are database-owned. Service routes, UI handlers, MCP helpers, and benchmark harnesses must not build their own logged-data projection caches and must not open SQLite directly. They may own query intent (for example the fields a route needs), but they call the logger DB object to execute it. The logger DB object owns connection threads, `mem`/disk table layout, write buffering, flush, reload-from-disk behavior, WAL tuning, and future FTS5/search tables. Do not move route-specific SQL into `DbWriter` or turn the DB layer into a pile of route helper methods such as `stats_detail_payload()` just to hide SQL. The boundary is execution and storage mechanics: ```rust db.ready().await?; db.query(sql, params).await?; db.write(event).await?; ``` `db.write(event).await` means the DB object accepted the event into its producer buffer. Tests that assert read-after-write rows must use the DB flush barrier or shutdown/reopen. Do not paper over visibility with sleeps, route projections, or direct SQLite readers. Empty table means empty result. Missing table or column means the schema contract is broken and must fail loudly; never add compatibility branches that treat missing ledger shape as empty data. Regression tests must guard the boundary. If a route needs ledger data, add a test that proves the route uses the DB object and a source guard that rejects raw `rusqlite` opens, direct `DbReader::open`, and service-owned projection state in production route code. Add a companion guard that prevents route-specific DB writer methods or missing-schema fallbacks from being introduced. ## Mock server boundary `crates/capsem-mock-server` is the single reusable local fixture server for benchmarks, doctor, protocol recording/replay, gateway/integration tests, and Ironbank. It owns mock protocol responses and deterministic local upstream behavior. Tests may launch it through `build_system/scripts/test/mock_server.py`, `tests/helpers/mock_server.py`, or `CAPSEM_MOCK_SERVER_BASE_URL`. Do not add another local HTTP/MCP/OAuth/model mock server for a feature. Extend the shared mock server and its fixtures instead, then assert the route through the relevant black-box test. ## Parallel VM and process lifecycle tests Read [parallel-lifecycle-tests.md](references/parallel-lifecycle-tests.md) before changing VM concurrency, companion supervision, or Apple VZ lifecycle tests. ## Adversarial testing Capsem is a security product. Every security-relevant feature needs tests that actively try to break invariants. Think like an attacker: - Can a corp-blocked domain be snuck through another provider's list? - Does an overlapping wildcard in allow+block always deny? - Does malformed input (empty strings, unicode, huge payloads, invalid JSON) get rejected? - Can path traversal escape the VirtioFS sandbox? - Can a guest process modify its own binaries? Stress-test boundary conditions. Write tests for the attacks you'd attempt yourself. ### Security invariants to verify in tests When touching security-relevant code, check these invariants have test coverage: | Invariant | What to test | Where | |-----------|-------------|-------| | VirtioFS share is `guest/` only | `session_dir/guest/` exists, symlinks resolve, host-only files (`session.db`, `serial.log`) are outside the share | `capsem-core::lib::tests` | | UDS sockets are 0600 | After bind, verify permissions exclude other users | `capsem-process` | | Process env is cleared | `env_clear()` called, only allowlisted vars passed | `capsem-service` spawn tests | | No `process::exit` on guest I/O | Control channel close causes loop break, not exit | `capsem-process` | | Sensitive logs are 0600 | `serial.log` created with restricted permissions | `capsem-process` | | Gateway auth on all routes | Every route except `GET /` returns 401 without token | `capsem-gateway::auth::tests` | | Auth rate limiting | 429 after threshold, resets after window | `capsem-gateway::auth::tests` | | CORS rejects external origins | Only localhost/127.0.0.1/tauri allowed | `capsem-gateway::tests` | | Body size limit | 413 for >10MB payloads | `capsem-gateway::proxy::tests` | | VM ID validation | Path traversal (`../`), dots, spaces, null bytes rejected | `capsem-gateway::terminal::tests` | | Rootfs read-only | runtime rootfs asset mounted ro, guest binaries 555 | `capsem-doctor` in-VM tests | | Suspend reports errors | IPC failure and timeout both return 500, not silent success | `capsem-service` tests | ## Test fixture anti-pattern: masking races with polling If all test fixtures wait/poll before asserting, the tests will never catch server-side race conditions. For every endpoint that talks to a VM socket, write at least one test that calls it IMMEDIATELY after provision (no `wait_exec_ready`, no `ready_vm` fixture). The server must handle readiness internally. **Pattern to avoid** (masks the bug -- server never needs wait logic because client always waits): ``` fixture calls provision -> fixture polls wait_exec_ready -> test calls exec ``` **Required test pattern** (catches the bug -- if server doesn't wait, test fails): ``` test calls provision -> test immediately calls exec -> server handles wait ``` See `tests/capsem-service/test_svc_exec_ready.py` for the regression tests that enforce this. ### wait_exec_ready is a single call, not a loop `wait_exec_ready` (in `tests/helpers/service.py`, `tests/helpers/mcp.py`, `tests/capsem-gateway/test_gw_e2e.py`) makes one exec call with the server-side timeout passed through. The server's `handle_exec` calls `wait_for_vm_ready` internally, which polls until the VM is ready. Do NOT add client-side retry loops -- that creates a double-wait where each retry can block for the full server timeout (30s client retries x 30s server wait = pathological cascade). One wait, one place. ### Exec latency regression gate `tests/capsem-serial/test_boot_timing.py::test_exec_latency_within_gate` asserts that provision-to-first-exec completes within `EXEC_LATENCY_GATE`. If this test fails, investigate boot time (process.log boot_timeline spans), not the wait mechanism. ## Where tests live - **Rust unit: sibling `tests.rs` file, not inline `mod tests { ... }`.** See the next subsection. - Rust integration: `crates/capsem-core/tests/` - In-VM diagnostics: `guest/artifacts/diagnostics/test_*.py` (see dev-testing-vm) - Hypervisor: KVM + Apple VZ tests (see dev-testing-hypervisor) - Frontend: `web/app/src/lib/__tests__/` (see dev-testing-frontend) - Python (builder): `tests/test_*.py` - Python integration (service daemon): `tests/capsem-*/` directories, each with its own conftest.py and pytest marker - Ironbank release ledger: `tests/ironbank/` (black-box only; no Rust implementation-derived expectations) ### Rust unit tests: sibling `tests.rs` pattern **Every Rust module keeps its unit tests in a sibling `tests.rs`, not an inline `mod tests { ... }` block.** The parent module declares: ```rust // foo.rs OR foo/mod.rs // ... production code ... #[cfg(test)] mod tests; ``` and the tests go in `tests.rs` in the same directory: ```rust // tests.rs -- sibling of foo.rs or child of foo/ use super::*; #[test] fn roundtrip() { ... } ``` **Why.** Inline `#[cfg(test)] mod tests { ... }` blocks are appended at the bottom of prod files and commonly hit 50–99% of the file's line count. That means every Read, grep, and scroll to reach production code walks past thousands of test lines first. Several modules in this codebase hit 4,000+ lines that way before extraction. Agents and humans both read faster when prod code isn't buried. **Mechanics.** - `tests.rs` is a submodule of the parent file -- `use super::*;` works, private items are visible, `#[cfg(test)]` on the `mod tests;` declaration still gates compilation. - For files that don't yet have a sibling directory (e.g. `lib.rs`, `foo.rs`), put `tests.rs` next to them in the same `src/` directory. - For files that are already `foo/mod.rs`, put `tests.rs` inside `foo/`.