dev-rust-patterns

# Rust Patterns ## Async / non-blocking Capsem uses tokio for all async I/O. The MITM proxy, vsock manager, and file monitor are all async. ### Never block the tokio runtime Long-running synchronous work (FUSE request processing, disk I/O, compression) must run on a dedicated thread via `tokio::task::spawn_blocking` or a dedicated `std::thread`. Blocking inside a tokio task starves other tasks. The VirtioFS FUSE server runs on its own thread for this reason -- FUSE ops are synchronous by nature (read, write, lookup) and can't be made async without significant complexity. ### Blocking-in-async anti-pattern (systemic -- audit, don't spot-fix) Any code path that does blocking I/O inside an async function or while holding a `tokio::sync::Mutex` is a bug. This causes the tokio worker thread to stall, freezing the entire gateway, UI, or network stack until the blocking operation completes. **What counts as blocking I/O:** - `std::process::Command` (subprocess execution) - `std::fs::*` (read, write, copy, remove_dir_all, create_dir_all) - `walkdir::WalkDir` (directory traversal) - `blake3::Hasher` on large data (hash computation) - `std::thread::sleep` **The fix pattern** -- in `capsem-service`, route handlers go through `ServiceState::off_worker` (`crates/capsem-service/src/blocking.rs`), and `crates/capsem-service/src/tests/async_io_contract.rs` refuses direct blocking calls. Elsewhere, the shape is: ```rust let result = tokio::task::spawn_blocking(move || { let rt = tokio::runtime::Handle::current(); rt.block_on(async { let mut guard = mutex.lock().await; sync_blocking_work(&mut guard) }) }).await.unwrap_or_else(|e| /* handle panic */); ``` **Example site:** the fork clone (`crates/capsem-service/src/vm_files/fork.rs`) runs `clone_sandbox_state` -- fsync, clonefile, and a full tree walk -- on the blocking pool. Any new tree copy, hash, or directory walk reachable from an async handler needs the same treatment. ### Channel patterns - `tokio::sync::mpsc` for producer-consumer (vsock data flow, telemetry events) - `tokio::sync::broadcast` for fan-out (serial output to multiple subscribers) - `tokio::sync::oneshot` for single-response request-reply (control messages) ### Logger DB boundary `capsem-logger` owns SQLite connection/thread/storage mechanics. Rust service, gateway, MCP, UI, and benchmark code must not call `rusqlite::Connection::open` or `DbReader::open` directly for telemetry/security ledgers, and must not add service-owned projection caches. Callers may own query intent, but the DB object owns execution: ```rust db.ready().await?; db.query(sql, params).await?; db.write(event).await?; ``` `db.write(event).await` is an accept boundary: the DB-owned producer buffer has taken responsibility for the event. Read-after-write tests must use the DB flush barrier or shutdown/reopen before asserting route-visible rows. Never add route sleeps, route projection caches, or caller-owned SQLite reads to force visibility. Do not hide route SQL by adding route-specific helpers to `DbWriter`; the DB writer is not a product route registry. Put connection threads, `mem`/disk tables, batching, flushing, WAL tuning, and future FTS5/search in the DB layer. Empty tables are fine. Missing tables or columns are schema contract failures and must not be converted into empty results. ### Coalescing buffer Terminal output uses a `CoalesceBuffer` (8ms window, 64KB cap) to batch small vsock reads into larger writes. This prevents xterm.js from choking on thousands of tiny updates. The pattern: accumulate into a buffer, flush on timer or size threshold. ### Graceful shutdown Use `tokio::select!` with a cancellation token or shutdown signal. Every long-running task must respect shutdown. Dangling tasks after VM exit cause resource leaks. ## Cross-compilation Guest binaries target `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`. Key gotchas: - **Platform-specific types**: `libc::ioctl` request param is `c_ulong` on macOS but `c_int` on Linux. Use `as _` to let the compiler infer the correct type. - **Linker**: `.cargo/config.toml` sets `linker = "rust-lld"` for both musl targets. - **No std dependencies**: musl builds are fully static. Avoid crates that link to system libraries. - **Test on both**: `cargo check --target aarch64-unknown-linux-musl` catches cross-compile errors without needing to boot a VM. ## Error handling - Use `anyhow::Result` for application code (capsem-app, scripts) - Use `thiserror` for library errors in capsem-core (typed, matchable) - Propagate errors up, don't swallow them. If a function returns `Result`, the caller must handle it. - Log errors at the point where you have context, then propagate. Don't log AND propagate (causes duplicate log lines). ## Host Unix boundary Reusable host Unix mechanics belong in the small modules under `capsem_foundation::unix`; application crates consume their typed APIs instead of calling `nix` or `libc` directly. This keeps ownership, EINTR handling, errno classification, CLOEXEC, no-follow behavior, and race checks identical across the CLI, service, process, guard, credentials, and core. - Accept `BorrowedFd` and return `OwnedFd` or `File`. Never return a raw descriptor whose close responsibility is implicit, and never pair `ManuallyDrop<File>` with a manual `close`. - Expected races are typed outcomes: a missing process and lock contention are not generic errors. Keep expected disconnects quiet; log unexpected failures once where the caller knows the operation, object identity, and errno. - Use `unix::contained::ContainedDir` for guest-influenced paths. Its descriptor-relative traversal, constrained open options, and regular-file checks must remain in the same syscall sequence; do not put a path-based check in front of a separate path-based action. - Use `unix::fs` for owner-only directories, no-follow regular-file access, filesystem capacity, atomic private publication, permission bits (`set_mode`), durability (`sync`, `sync_before_barrier`, `sync_filesystem`) and copy-on-write cloning (`clone_file_into`, `copy_sparse`). Use `unix::tree_clone` to copy a directory tree a guest can write. Use `unix::lock` for advisory locks, including synchronous read-modify-write sections. - When foundation lacks a primitive, add it to foundation with its tests; never reach for `nix` or `libc` at the call site "just this once". - Inside foundation, `nix` is the syscall layer and `unix::errno::io` converts its errors. A raw `libc::` call is allowed only where `nix` cannot express it (no wrapper, or a platform gap such as `MSG_NOSIGNAL` on macOS), and each one is listed with that reason in `tests/citadel/unix_boundary_debt.toml`. - Put each primitive in its own module with tests in the sibling `tests.rs`. Cover errno preservation, ownership after the source is dropped, symlink and path-replacement races, special files, exact modes, and concurrent callers. Direct Unix calls remain valid only in modules that implement a kernel or platform ABI rather than reusable host policy, such as KVM/FUSE and guest audit plumbing. `tests/citadel/test_unix_boundary.py` and its exact domain-ABI inventory enforce that distinction; extending the inventory requires explicit review, not a convenience exemption. The same guard inventories every raw `libc::` call and `extern "C"` binding inside foundation itself, rejects a new one that `nix` already wraps, and catches aliased imports (`use ::libc as sys`), `extern "C" { fn ... }` blocks and renamed or table-form manifest dependencies. Before it did, foundation's own internals were unscanned and raw calls with `nix` wrappers accumulated there unnoticed. ## Bidirectional I/O -- thread per direction When bridging two blocking file descriptors bidirectionally (e.g., TCP socket to vsock in `net_proxy.rs`, or master PTY to vsock in `capsem-pty-agent`), doing both reads and writes in a single thread using `poll(2)` causes deadlocks. If both outgoing buffers fill simultaneously, a single thread blocks on writing and stops reading, creating mutual lockup. Always spawn a dedicated thread for at least one direction (`std::thread::spawn` for `fd_b -> fd_a` while the main thread handles `fd_a -> fd_b`). ## Serde -- avoid `serde_json::Value` on LLM payloads The MITM proxy and ai_traffic parsers handle massive HTTP payloads (megabytes of tool calls, histories, images). Parsing these into `serde_json::Value` does full DOM allocation, which is inefficient and risks memory exhaustion. **Rules:** - Define targeted structs with `#[derive(Deserialize)]`. Serde skips and discards fields not in the struct without allocating memory for them. - For struct fields that hold large, unconstrained JSON (tool call arguments, function responses, full model outputs) and are only converted to strings: use `Box<serde_json::value::RawValue>` instead of `serde_json::Value`. `RawValue` keeps the JSON as an unparsed string slice -- zero DOM allocation. Access the raw JSON string via `.get()`. - Never add `serde_json::Value` fields to structs that parse LLM request/response bodies. If you only need a string representation, use `RawValue`. If you need to traverse nested fields, use a typed struct. - Remove unused fields from deserialization structs -- they still force Serde to allocate. **Example -- before (bad):** ```rust struct FunctionCall { name: Option<String>, args: Option<serde_json::Value>, // full DOM parse of potentially huge args } // later: let arguments = fc.args.as_ref().map(|v| v.to_string()); ``` **After (good):** ```rust struct FunctionCall { name: Option<String>, args: Option<Box<serde_json::value::RawValue>>, // zero-copy string slice } // later: let arguments = fc.args.as_ref().map(|v| v.get().to_owned()); ``` ## Memory and resource management - **File handle limits**: VirtioFS caps at 4096 open file handles, returns `EMFILE` beyond that. - **Read size limits**: VirtioFS clamps reads to 1MB, gather buffers to 2MB. - **Safe deserialization**: `read_struct` returns `Option<T>` with bounds checks in all builds (not just debug). - **irqfd for interrupt delivery**: Guest interrupt signaling uses `irqfd` to avoid cross-thread syscall overhead. ## Concurrency patterns - **RwLock for caches**: Cert authority uses `RwLock<HashMap>` -- many readers, rare writers. Use `read()` first, upgrade to `write()` only on cache miss. - **Arc for shared state**: VM state, proxy config, and telemetry handles are `Arc`-wrapped for sharing across tasks. - **Per-connection tasks**: The MITM proxy spawns a new tokio task per connection. Each task owns its TLS state and upstream connection. No shared mutable state between connections. ### Host-serialization locks for per-host critical sections When a service orchestrates N sibling child processes on a single host and some operations cannot safely run two-at-a-time on that host -- whether because of a framework constraint (Apple VZ save/restore) or because of shared-resource starvation (VZ teardown + WAL checkpoint + virtiofs drain all competing for main-thread and I/O bandwidth) -- park a `tokio::sync::Mutex<()>` on the service's shared state struct and acquire it at the top of the handler for the whole duration of the critical section. `Mutex<()>` isn't a weird construction: the unit value is the lock-token, the type signals "pure serialization, no protected payload". `Semaphore::new(1)` is equivalent -- pick one and stay consistent. Current instances in `crates/capsem-service/src/main.rs`: - **`save_restore_lock`**: serializes Apple VZ `saveMachineStateToURL` / `restoreMachineStateFromURL` across sibling VMs. Concurrent save/restore corrupts the VirtioFS ring state on the unlucky VM, surfaces as ext4-on-loop0 I/O errors after resume. Held through `handle_suspend` (IPC + child-exit wait) and `handle_resume` (spawn + `wait_for_vm_ready`). See `web/docs/src/content/docs/gotchas/concurrent-suspend-resume.md`. - **`shutdown_lock`**: serializes VM teardown across `handle_delete` / `handle_stop` / `handle_purge` / `handle_run`. Without it, N concurrent deletes under load starve each other of the bandwidth each `capsem-process` needs to exit cleanly within the 1s fast-path budget; past the budget the service SIGKILLs mid-checkpoint and leaves a non-empty `session.db-wal`. Held through `shutdown_vm_process` for the whole `SIGTERM` + `wait_for_process_exit` window. When to reach for this pattern: - Symptom is "works solo, fails under concurrency on the same host." - Root cause is a *per-host* resource, not per-VM: Apple VZ main thread, virtiofsd, DbWriter checkpoint, APFS fsync. - Production runs exactly one service per host per user, so an in-process tokio mutex is enough -- no need for a file-lock or distributed primitive. When NOT to reach for it: - If the contention is per-VM (two handlers acting on the same VM), protect the VM entry in `instances: Mutex<HashMap<...>>` instead. - If the "contention" is really a durability race (writer thread hasn't flushed), the right fix is usually the signal-handler explicit-cleanup pattern below, not another serialization lock. ### Signal-driven explicit cleanup for background-thread owners Any long-running Rust process that owns background threads (SQLite writer, the fs-monitor scan loop, MCP aggregator subprocess, vsock relay) and runs under a bounded SIGTERM-to-SIGKILL budget must NOT rely on `Drop` + tokio-runtime-drop ordering to finish cleanup. On SIGTERM, hand owned resources to the signal handler and drain them synchronously BEFORE letting the main run loop return. Symptom when this is missing: under concurrent teardowns on one host, the service SIGKILLs a child mid-checkpoint or mid-flush. Visible as `session.db-wal` left non-empty, missing `fs_events` rows, dangling aggregator subprocesses. Works solo, fails under `-n 4`. Concrete primitives in this tree: - **`DbWriter::shutdown_blocking(&self)`** — takes the stored mpsc sender, joins the writer thread, runs the final `PRAGMA wal_checkpoint(TRUNCATE)`. Arc-safe: other `Arc<DbWriter>` clones remain valid but their writes become no-ops. Idempotent. Drop delegates to it. - **`FsMonitor::shutdown_and_join(&self)`** — sends on the shutdown channel so the scan loop runs one final scan and emission, then joins the thread. Bounded at two scans and two emissions. Must run BEFORE DbWriter shutdown, because fs_events fan into DbWriter. - **`CAPSEM_TEST_SLOW_CHECKPOINT_MS`** — test-only env var in `writer_loop` that inserts a sleep before the final checkpoint. Use in tests that need to distinguish explicit cleanup from implicit runtime-drop ordering. Canonical wiring in `crates/capsem-process/src/main.rs`: ```rust struct Shutdown { db: Option<Arc<DbWriter>>, fs_monitor: Option<FsMonitor>, } impl Shutdown { fn drain_blocking(&mut self) { // fs_events fan into DbWriter -- flush fs_monitor first. if let Some(m) = self.fs_monitor.take() { m.shutdown_and_join(); } if let Some(db) = self.db.take() { db.shutdown_blocking(); } } } // Populate as owners are constructed: shutdown.lock().await.db = Some(Arc::clone(&db)); shutdown.lock().await.fs_monitor = Some(monitor); // Signal handler drains through spawn_blocking, then stops the run loop: rt.spawn(async move { /* wait on SIGTERM/SIGINT */ let mut owned = std::mem::take(&mut *shutdown.lock().await); let _ = tokio::task::spawn_blocking(move || owned.drain_blocking()).await; unsafe { core_foundation_sys::runloop::CFRunLoopStop(...); } }); ``` Key properties: 1. **Deterministic order.** The drain order is explicit (fs_monitor -> db), not "whatever reverse-declaration-order Drop happens to give us after tokio aborts tasks." 2. **Synchronous join.** The handler waits for each background thread to finish. No "hope the task finishes before the runtime drops." 3. **Run loop stops last.** `CFRunLoopStop` (macOS) fires only after drain returns. Main returns afterwards; the remaining tokio-runtime drop is now a no-op fast path because the heavy work already completed. 4. **Arc-safe shutdown APIs.** `shutdown_blocking(&self)` works through a shared `Arc<DbWriter>` — callers don't have to chase down every clone. Use `std::sync::Mutex<Option<Sender>>` internally; the hot-path `write()` clones the sender under the lock and releases it before `.await`. When to reach for this pattern: - The process has `std::thread::spawn` or `tokio::task::spawn_blocking` workers that run durability-critical work on shutdown (WAL checkpoint, queue flush, child-process wait). - A parent sends SIGTERM then SIGKILLs after a short, fixed budget. - Today's cleanup relies on Drop running inside tokio task abort — i.e., you can't draw a line between "cleanup finished" and "run loop exited." Call out when NOT to use it: - One-shot CLIs that exit on natural task completion (no run loop, no signal window). - Workers whose only side effects are in-memory (no durability to lose). When adding a new long-running process or a new background-thread owner, wire it through `Shutdown` from day one. Don't ship a new binary that "should be fine because Drop will run" — under load, Drop won't run in time. ## Logging - `tracing` crate with `FmtSpan::CLOSE` for timing spans - `RUST_LOG=capsem=debug` for full boot timing breakdown - `RUST_LOG=capsem=info` for top-level only