dev-testing-vm

# In-VM Testing ## Manual debug session lifetime A bounded command's `run-<pid>` scratch is reclaimed when its owner exits. `ServiceInstance.stop(cleanup=False)` does not retain that outer generation. Never promise to preserve a signed-in VM there across a runtime restart. For a long-lived manual session, ask `load_paths()` for the `test-temp` stage, create a separate managed `run-*` generation, and hold its policy lease through `cache.leases.retain_path`. Put the service home inside that generation and record its owner, lease, source and stopfile in Sprinty. Before a handoff, the replacement owner must acquire that same lease before the old owner releases it. Preserve the named VM through its public lifecycle API. Verify the home still exists after the old wrapper exits; only then claim it was retained. Keep one active manual runtime and stop superseded owners by exact stopfiles. Debug fixtures set `CAPSEM_TRAY_HEADLESS=1` to avoid extra menu-bar icons. ## capsem-doctor The diagnostic suite runs inside the guest VM via pytest. Tests live in `guest/artifacts/diagnostics/` and are baked into the rootfs. ### Running diagnostics ```bash just exec "capsem-doctor" # Full suite (~10s total) just exec "capsem-doctor -k sandbox" # Only sandbox tests just exec "capsem-doctor -k network" # Only network tests just exec "capsem-doctor -x" # Stop on first failure ``` ### Test categories | File | What it verifies | |------|------------------| | `test_sandbox.py` | Read-only rootfs, binary permissions, setuid/setgid, kernel hardening (no modules, no debugfs, no IPv6, no swap), process integrity, network isolation (dummy0, fake DNS, iptables) | | `test_network.py` | MITM CA in system store, curl without -k, Python urllib HTTPS, CA env vars, HTTP/80 blocked, non-443 blocked, direct IP blocked, multi-domain DNS, AI provider domains | | `test_environment.py` | TERM/HOME/PATH env vars, bash shell, kernel version, aarch64 arch, mount points, tmpfs | | `test_runtimes.py` | python3/pip3 versions, hermetic pip install into the venv, hermetic apt install of a local .deb, apt sandbox TLS trust, Python file I/O | | `test_workflows.py` | Text write/read, JSON roundtrip, shell pipes, large file (10MB) | | `test_virtiofs.py` | VirtioFS mount, ext4 loopback, workspace I/O, pip install, file delete+recreate | The runtime rootfs is minimal: no node, uv, git or AI CLIs. Those live in the OCI images under `images/`, and `tests/images/` proves each one boots and runs its agent. A host-side test that needs a tool the runtime lacks -- a test runner, network tool, package manager, model SDK, agent CLI or benchmark peer -- opens a `capsem-debug` session (`tests/helpers/debug_session.py`) and execs into its workload; `"target": "vm"` still reaches the VM. Never add test tooling to `runtime_apt_packages`. The image, its digest pin and how to rebuild it are in `tests/fixtures/oci/README.md`. ### Adding new in-VM tests 1. Add test functions to the appropriate `guest/artifacts/diagnostics/test_*.py` or create `test_<category>.py` 2. Use `from conftest import run` for shell commands, `output_dir` fixture for temp files 3. Tests auto-skip outside the capsem VM (conftest checks for root + writable /root) 4. Rebuild rootfs with `just _build-assets` to bake new test files into the image 5. For fast iteration during development, tests in `diagnostics/` are also repacked into the initrd by `just exec`, so `just exec "capsem-doctor"` picks up changes without a full rootfs rebuild 6. Verify: `just exec "capsem-doctor -k <your_test>"` ## Session inspection For a live or retained product session, inspect telemetry through the typed service routes (`capsem_timeline`, history, statistics, and security tools in `@capsem/mcp`). Direct database inspection is an offline diagnostic for a coherent, quiesced evidence copy after its ledger worker has stopped: ```bash uv run --project build_system --frozen python build_system/scripts/doctor/check_session.py --list uv run --project build_system --frozen python build_system/scripts/doctor/check_session.py --db /path/to/evidence/session.db uv run --project build_system --frozen python build_system/scripts/doctor/check_session.py --db /path/to/evidence/session.db -n 10 ``` Checks: session ledgers exist (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events), row counts, orphaned tool_calls, AI-provider consistency. ## Verifying telemetry pipelines Each pipeline can be tested with a targeted VM command: - **fs_events**: `just exec 'touch /root/test.txt && sleep 1'` then inspect the typed timeline route - **net_events**: `just exec 'curl -s https://api.anthropic.com/ && sleep 1'` - **model_calls/tool_calls**: boot interactively, run `claude -p "what is 2+2"` - **MCP-origin tool_calls**: boot interactively, run `claude -p "use fetch to get https://example.com"`, then inspect the typed timeline route for MCP-origin evidence If events are missing: check boot logs for daemon startup, vsock connection acceptance, and whether the VM lived long enough for the debouncer to flush (add `sleep 1`). ## Test fixture The fixture (`tests/fixtures/session/test.db`) is a real session DB shared by frontend mock mode and Rust roundtrip tests. No synthetic data. ### Updating the fixture ```bash # 1. Run integration test to generate a rich session python3 build_system/scripts/test/integration_test.py --binary cache/target/cargo/debug/capsem --assets assets # 2. Stop the fixture owner, preserve a coherent ledger/archive copy, and # inspect completeness uv run --project build_system --frozen python build_system/scripts/doctor/check_session.py --db /path/to/evidence/session.db # 3. Fixture refresh has no Just convenience command. Copy, checkpoint, and # scrub the selected DB using the procedure in /dev-session-debug. # 4. Verify cargo test --workspace ``` The fixture must contain: both allowed and denied net_events, created/modified/deleted fs_events, model_calls with cost > 0, tool_calls with origin populated.