dev-debugging

# Debugging ## The rule Never fix code before you understand why it broke. The temptation to "just make the test pass" or "just patch the symptom" leads to fragile fixes that hide deeper problems. Follow the three-step workflow below every time. ## Choose the cheapest truthful feedback loop Capsem now has two distinct execution surfaces: - Just exposes the small public product interface. - Python under `src/capsem/gate/` owns build/test/release plans, graph edges, locking, resources, evidence, and cleanup. For a gate failure, debug the Python step and action that failed; do not infer the implementation from the Just recipe or recreate it with shell commands. Start with: ```bash uv run --project build_system --frozen capsem-gate runs last --failed uv run --project build_system --frozen capsem-gate <command> --dry-run uv run --project build_system --frozen capsem-gate <command> --graph ``` Use the smallest focused pytest, cargo, pnpm, or script command for red/green work. Use `just focus-test functional` for focused integration feedback, and `just fast-test` for the fast gate itself. Use `just test` when the forward fix is ready for the complete local verification a release requires. Direct diagnostics which may block, build, launch children, or wait on input must use the portable bounded-process wrapper rather than a bare shell command: ```bash python3 build_system/scripts/ci/run-bounded-command.py --timeout-seconds 1800 -- <command> <args...> ``` The wrapper closes stdin, creates a dedicated process group, and terminates the whole group on timeout or interruption. This prevents a PTY-backed `docker build -f -`, compiler, test runner, or child helper from surviving its owning diagnostic. Pick a finite timeout appropriate to the focused operation; do not wrap `just test` or a release command, whose config-owned step timeouts, journal, teardown, and resumable graph are the authority. ### Diagnostic continuation for a late gate failure When a clean candidate fails after expensive predecessors have succeeded, use **diagnostic continuation** to reuse that retained state and reach the frontier quickly. The current CLI retains legacy flag names: ```bash uv run --project build_system --frozen capsem-gate runs last --failed uv run --project build_system --frozen capsem-gate candidate --prefix <retained-prefix> --from <failed-step> ``` The `--from` step runs; graph predecessors are reported as `carried`. Before using it, match the prefix and failed step to the origin run, confirm the required predecessor really completed there, and record the origin run ID in the diagnosis. Prefer a fresh focused test when the edit invalidates the expensive producer itself. Carried steps with external runtime products are revalidated before resumed work. If one was reclaimed, use the exact owning `--from` label in the refusal; do not guess an earlier frontier or recreate the product by hand. Call the result `diagnostic-passed` or `diagnostic-failed`, even if the current implementation still summarizes a zero exit as `ok`. It combines earlier outputs with current source and therefore answers only whether the new segment can proceed. It does not prove the complete current tree. Never use diagnostic continuation with `release-binaries` or `release-profile`, and never let it authorize publication. After the fix, use the smallest owning `focus-test` group. Run `just test <commit>` when complete local whole-system proof is useful. It is not required before release: the hosted release lane owns qualification and never consumes the local journal. ## Step 1: Reproduce with a test Before touching any implementation code, write a test that captures the bug. This test must: - Fail right now, demonstrating the broken behavior - Be specific enough to distinguish the bug from correct behavior - Live in the right test location (see dev-testing for where tests go) If you can't reproduce it in a test, you don't understand it well enough to fix it. For VM-level issues, use capsem-doctor or write a targeted diagnostic command: ```bash just exec "<command that triggers the bug>" ``` For telemetry issues, use session inspection: ```bash python3 build_system/scripts/doctor/check_session.py ``` ## Step 2: Diagnose the root cause With a failing test in hand, investigate. Do not skip this step. Common diagnostic approaches: **MCP triage trio (run FIRST when an investigation is open-ended):** ``` capsem_panics { since: "1h" } # any Rust panic in any host log? -> rank highest capsem_triage { id: "vm-1" } # ranked recent ipc-warns + 4xx/5xx + slow_ops + session.db errors capsem_timeline { id: "vm-1", trace_id: "<X>" } # follow ONE logical operation across exec/tool/net/fs/model ``` These read post-W2 JSON logs (`~/.capsem/run/{service,mcp,gateway,tray}.log` + capsem-app's latest jsonl) and post-W6 session.db tables. The W4 `target=fs op=fsync duration_ms=...` markers feed `capsem_triage`'s slow-op rank; the W3 schema_hash check appears in `capsem_panics` output as `IPC handshake failed; refusing connection` events. Always start with `capsem_panics` -- a single panic outranks a hundred warns. **Cross-version mix?** The `service.start` log line emits `protocol_version=N, schema_hash=<hex>` per binary. If the support bundle (`capsem support-bundle`) shows two different schema_hash values across binaries, you're hitting the W3 cross-version-mix detection -- rebuild + restart the lagging binary. **Integration-test failures: read the preserved service log.** When any integration test fails, the test fixture (`tests/helpers/service.py::ServiceInstance`, the e2e `RealService`, and the MCP `_start_capsem_service`) archives its tmp_dir to `cache/target/tests/evidence/<timestamp>-<worker>-<nodeid>/<tmp-basename>/` **before** the usual rmtree. The failing test's stderr has the exact path: look for a line `ARTIFACT: preserved /var/folders/... -> cache/target/tests/evidence/...`. Inside that directory: ``` service.log host-side capsem-service debug log (RUST_LOG=debug) logs/gateway.log gateway stdout/stderr logs/tray.log tray stdout/stderr (if spawned) sessions/<vm-id>/process.log per-VM capsem-process log (vsock bridge, IPC, spawn chain) sessions/<vm-id>/serial.log VM serial console (kernel boot, capsem-init, agent startup) sessions/<vm-id>/session.db SQLite telemetry DB (net_events, model_calls, ...) persistent/<name>/... persistent-VM state (checkpoint.vzsave, workspace) ``` `cache/target/tests/evidence/` is gitignored. Multiple failures sharing a session-scoped service land in different subdirs but the latest run's name tags them by the most recent failing nodeid. First place to look for "VM didn't become exec-ready" style failures: `sessions/<id>/serial.log` (did the VM boot?) and `sessions/<id>/process.log` (did the agent come up + IPC handshake?). For "provision hung" or service-side contention: `service.log`, grep for the VM id. **Rust code**: Read the code path the test exercises. Trace the data flow. Add `tracing` instrumentation if needed (`RUST_LOG=capsem=debug`). Check if the issue is in capsem-core, capsem-app, or capsem-agent. **Guest VM issues**: Boot with targeted commands and inspect behavior: ```bash just exec "capsem-doctor -k <category>" # Run specific diagnostic category just exec "<manual investigation command>" ``` Check boot logs for daemon startup failures, vsock connection issues, or timing problems. **Network/security issues**: Check the network intercept path -- SNI parsing, HTTP/DNS/model normalization, cert minting, `SecurityEvent` construction, security rule evaluation, plugin execution, runtime materialization, and ledger materialization. Do not debug by adding credential handling to formatters, routes, DB readers, frontend transforms, or harnesses. Use session DB to see what actually happened: ```bash python3 build_system/scripts/doctor/check_session.py # Check net_events for domain, decision, status_code ``` **Frontend issues**: Run `just dev ui`, open Chrome DevTools, check console errors, use `take_screenshot` to capture state. See dev-testing-frontend for the full visual verification workflow. **Build pipeline issues**: Check `cache/containers/logs/build.log` -- all build infrastructure (runner, code signing, generation scripts) logs here. The runner (`build_system/packaging/macos/run_signed.sh`) and `_generate-settings` recipe both append to this file. Never write diagnostics to stdout from build scripts (it contaminates binary output like `mcp-export`). For a Python gate or release failure, inspect the recorded run first. Step labels, argv, timing, captured output, resource events, and the first failed action live under `cache/target/gate-runs/`; `capsem-gate runs last --failed` is the supported reader. `cache/containers/logs/build.log` is supporting build evidence, not the gate's execution ledger. Do not retry a release through its script, `capsem-admin release`, or a GitHub workflow. The public command first accepts the exact qualification journal, then runs prechecks, source publication, and dispatch as one graph. A direct script or workflow skips the edges that make release safe. **Telemetry pipeline issues**: The canonical session ledgers (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events) each have their own boundary. If a table is empty or has wrong data: - Check if the guest daemon started (boot logs) - Check if the vsock connection was accepted (host logs) - Check timing -- did the VM shut down before the DB-owned buffer flushed? Use the DB flush barrier or shutdown/reopen proof; do not add a sleep or poll. For route latency or stale stats, do not add service-owned logged-data projections. Logged-data hot state belongs inside the logger DB object as table-level `mem`/disk ownership with DB-layer tests and benchmarks. Service routes may describe the query they need, but production service code must not open rusqlite connections or `DbReader` directly. Do not "fix" route latency by hardcoding route-specific query helpers in `DbWriter`, by adding service caches, or by swallowing missing tables/columns as empty data. The correct diagnosis target is the DB object: connection/thread ownership, `mem`/disk layout, batching, flush, and query execution. If the schema is missing, surface the broken ledger contract. Write down what you find. The diagnosis should explain *why* the bug exists, not just *where* the symptom appears. ## Concurrency flakes are product bugs, not test-tuning problems `just test` runs the python suite under `pytest -n 4 --dist=loadfile`. Four real VMs boot in parallel; this is dogfooding. Capsem ships as a multi-VM sandbox for AI agents -- if the test suite cannot safely run 4 concurrent VMs, real users running an agent farm will hit the same bug. When a test flakes only under concurrency, the diagnosis target is **Capsem's product code**, not the test: - "Suspend timed out" appearing only at `-n 4` -> `handle_suspend` IPC race; investigate the `with_quiescence` path and the `Suspend` round-trip, not the test timeout - "Session did not become ready" only with multiple parallel provisions -> Apple VZ resource contention, VirtioFS lock, or service handle_provision serialization gap - Two tests collide on the same VM/session name -> `validate_vm_name` / persistent registry has a TOCTOU; UUID prefix in the test is not the bug - "Connection refused" on a per-VM UDS only at `-n 4` -> service spawned the process but didn't wait for the socket to be bound; race in the spawn path - A test passes serial but hangs at n=4 -> a global lock somewhere (state mutex held across an await, blocking Tokio worker; or a sync `std::Mutex` on a hot path) Anti-patterns to avoid: - Adding `time.sleep` in the test "to let things settle" - Bumping a per-test timeout from 30s to 120s "because it's flaky" - Marking the test `serial` -- defeats the dogfooding signal - Adding retries with backoff in the client Right pattern: capture a service log of the failing run (set `RUST_LOG=capsem=trace`), find the operation that took unexpectedly long or returned an error, fix the underlying race in capsem-service / capsem-process / capsem-core. Then re-run at `-n 4` to confirm. ## Step 2.5: Fix the pattern, not the instance When diagnosis reveals a **systemic pattern** (the same mistake repeated across the codebase), the fix must cover every instance -- not just the one that was reported. - **Audit the entire codebase for the same pattern.** If blocking I/O in async context caused one hang, grep for every other site that does the same thing. A bug is a symptom -- the pattern is the disease. - **Never simplify a fix to the minimum diff.** A "quick fix" that patches one call site while 6 others have the identical problem is not a fix -- it's deferred breakage. - **Document the pattern in the relevant skill** (e.g., dev-rust-patterns) so it's never reintroduced. - **Add tests that would catch the pattern** if it recurs (e.g., a contract test between the frontend and backend response format). Example: an MCP tool hang was caused by blocking I/O (clonefile, walkdir, blake3) on tokio worker threads. The same anti-pattern existed in 7 sibling tool handlers, a background timer, and asset hash verification. Fixing only the reported call would have left 9 other sites broken. ## Step 3: Fix with a comprehensive solution Now that you understand the root cause, write the fix. The fix should: - Make your reproducing test pass - Not break any existing tests (`just test`) - Address the root cause, not just the symptom - Include the test from Step 1 in the same commit After the fix, run the full validation: 1. `just test` -- unit + cross-compile + frontend 2. `just exec "capsem-doctor"` -- VM smoke test 3. If the bug touched telemetry: `python3 build_system/scripts/doctor/check_session.py` after a real session A diagnostic continuation may shorten investigation before this validation; it never replaces any item in the final proof. ## Local/CI execution parity When a bug appears only in CI, first identify the exact production entrypoint, runner dependency, architecture, environment variable, permission, device, or service-manager difference that local testing skipped. Reproduce every portable Linux difference in Docker and execute the same production entrypoint or shared predicate that CI executes. A hand-written approximation is not a regression test. Keep an executable parity test after the fix and audit sibling workflows for the same one-sided assumption. If reproduction crosses an unavoidable platform boundary, document the boundary and preserve the nearest local contract plus the required owning release-job or physical-machine proof. Do not relabel an unreproduced CI failure as transient. ## What NOT to do - **Do not "fix" a failing test by changing the test assertion.** The test is telling you something. Listen to it. If the test is genuinely wrong, explain why in detail before changing it. - **Do not dismiss failures as "pre-existing" or "unrelated."** Investigate every failure. If it truly is pre-existing, file it and fix it -- don't leave broken windows. - **Do not guess-and-check.** Random changes hoping something sticks waste time and often introduce new bugs. Understand first, then act. - **Do not patch symptoms.** If requests fail because gzip content-encoding isn't handled, don't strip the Accept-Encoding header -- implement proper decompression. Fix the system, not the surface. - **Do not apply narrow fixes to systemic problems.** If the same anti-pattern exists in 7 places and you fix 1, you haven't fixed the bug -- you've hidden 6 more. Audit first, then fix all instances in a single pass.