dev-rust-patterns

name	dev-rust-patterns
description	Rust patterns and hard-won lessons. Use when writing Rust in capsem-core, capsem-app, or capsem-agent: async/tokio, non-blocking I/O, cross-compilation.

Rust Patterns

Async / non-blocking

Capsem uses tokio for all async I/O. The MITM proxy, vsock manager, file monitor, and auto-snapshot scheduler are all async.

Never block the tokio runtime

Long-running synchronous work (FUSE request processing, disk I/O, compression) must run on a dedicated thread via tokio::task::spawn_blocking or a dedicated std::thread. Blocking inside a tokio task starves other tasks.

The VirtioFS FUSE server runs on its own thread for this reason -- FUSE ops are synchronous by nature (read, write, lookup) and can't be made async without significant complexity.

Blocking-in-async anti-pattern (systemic -- audit, don't spot-fix)

Any code path that does blocking I/O inside an async function or while holding a tokio::sync::Mutex is a bug. This causes the tokio worker thread to stall, freezing the entire gateway, UI, or network stack until the blocking operation completes.

What counts as blocking I/O:

• std::process::Command (subprocess execution)
• std::fs::* (read, write, copy, remove_dir_all, create_dir_all)
• walkdir::WalkDir (directory traversal)
• blake3::Hasher on large data (hash computation)
• std::thread::sleep

The fix pattern -- same as call_mcp_tool in crates/capsem-app/src/commands/mcp.rs:

let result = tokio::task::spawn_blocking(move || {
    let rt = tokio::runtime::Handle::current();
    rt.block_on(async {
        let mut guard = mutex.lock().await;
        sync_blocking_work(&mut guard)
    })
}).await.unwrap_or_else(|e| /* handle panic */);

Known fixed sites (2026-03-27): MCP file tool dispatch, auto-snapshot timer (vsock_wiring.rs), asset hash verification (asset_manager.rs). If you add new file tools or snapshot operations, use the same spawn_blocking pattern.

Channel patterns

• tokio::sync::mpsc for producer-consumer (vsock data flow, telemetry events)
• tokio::sync::broadcast for fan-out (serial output to multiple subscribers)
• tokio::sync::oneshot for single-response request-reply (control messages)

Logger DB boundary

capsem-logger owns SQLite connection/thread/storage mechanics. Rust service, gateway, MCP, UI, and benchmark code must not call rusqlite::Connection::open or DbReader::open directly for telemetry/security ledgers, and must not add service-owned projection caches.

Callers may own query intent, but the DB object owns execution:

db.ready().await?;
db.query(sql, params).await?;
db.write(event).await?;

db.write(event).await is an accept boundary: the DB-owned producer buffer has taken responsibility for the event. Read-after-write tests must use the DB flush barrier or shutdown/reopen before asserting route-visible rows. Never add route sleeps, route projection caches, or caller-owned SQLite reads to force visibility.

Do not hide route SQL by adding route-specific helpers to DbWriter; the DB writer is not a product route registry. Put connection threads, mem/disk tables, batching, flushing, rehydration, WAL tuning, and future FTS5/search in the DB layer. Empty tables are fine. Missing tables or columns are schema contract failures and must not be converted into empty results.

Coalescing buffer

Terminal output uses a CoalesceBuffer (8ms window, 64KB cap) to batch small vsock reads into larger writes. This prevents xterm.js from choking on thousands of tiny updates. The pattern: accumulate into a buffer, flush on timer or size threshold.

Graceful shutdown

Use tokio::select! with a cancellation token or shutdown signal. Every long-running task must respect shutdown. Dangling tasks after VM exit cause resource leaks.

Cross-compilation

Guest binaries target aarch64-unknown-linux-musl and x86_64-unknown-linux-musl. Key gotchas:

• Platform-specific types: libc::ioctl request param is c_ulong on macOS but c_int on Linux. Use as _ to let the compiler infer the correct type.
• Linker: .cargo/config.toml sets linker = "rust-lld" for both musl targets.
• No std dependencies: musl builds are fully static. Avoid crates that link to system libraries.
• Test on both: cargo check --target aarch64-unknown-linux-musl catches cross-compile errors without needing to boot a VM.

Error handling

• Use anyhow::Result for application code (capsem-app, scripts)
• Use thiserror for library errors in capsem-core (typed, matchable)
• Propagate errors up, don't swallow them. If a function returns Result, the caller must handle it.
• Log errors at the point where you have context, then propagate. Don't log AND propagate (causes duplicate log lines).

Bidirectional I/O -- thread per direction

When bridging two blocking file descriptors bidirectionally (e.g., TCP socket to vsock in net_proxy.rs, or master PTY to vsock in capsem-pty-agent), doing both reads and writes in a single thread using poll(2) causes deadlocks. If both outgoing buffers fill simultaneously, a single thread blocks on writing and stops reading, creating mutual lockup. Always spawn a dedicated thread for at least one direction (std::thread::spawn for fd_b -> fd_a while the main thread handles fd_a -> fd_b).

Serde -- avoid serde_json::Value on LLM payloads

The MITM proxy and ai_traffic parsers handle massive HTTP payloads (megabytes of tool calls, histories, images). Parsing these into serde_json::Value does full DOM allocation, which is inefficient and risks memory exhaustion.

Rules:

• Define targeted structs with #[derive(Deserialize)]. Serde skips and discards fields not in the struct without allocating memory for them.
• For struct fields that hold large, unconstrained JSON (tool call arguments, function responses, full model outputs) and are only converted to strings: use Box<serde_json::value::RawValue> instead of serde_json::Value. RawValue keeps the JSON as an unparsed string slice -- zero DOM allocation. Access the raw JSON string via .get().
• Never add serde_json::Value fields to structs that parse LLM request/response bodies. If you only need a string representation, use RawValue. If you need to traverse nested fields, use a typed struct.
• Remove unused fields from deserialization structs -- they still force Serde to allocate.

Example -- before (bad):

struct FunctionCall {
    name: Option<String>,
    args: Option<serde_json::Value>,  // full DOM parse of potentially huge args
}
// later: let arguments = fc.args.as_ref().map(|v| v.to_string());

After (good):

struct FunctionCall {
    name: Option<String>,
    args: Option<Box<serde_json::value::RawValue>>,  // zero-copy string slice
}
// later: let arguments = fc.args.as_ref().map(|v| v.get().to_owned());

Memory and resource management

• File handle limits: VirtioFS caps at 4096 open file handles, returns EMFILE beyond that.
• Read size limits: VirtioFS clamps reads to 1MB, gather buffers to 2MB.
• Safe deserialization: read_struct returns Option<T> with bounds checks in all builds (not just debug).
• irqfd for interrupt delivery: Guest interrupt signaling uses irqfd to avoid cross-thread syscall overhead.

Concurrency patterns

• RwLock for caches: Cert authority uses RwLock<HashMap> -- many readers, rare writers. Use read() first, upgrade to write() only on cache miss.
• Arc for shared state: VM state, proxy config, and telemetry handles are Arc-wrapped for sharing across tasks.
• Per-connection tasks: The MITM proxy spawns a new tokio task per connection. Each task owns its TLS state and upstream connection. No shared mutable state between connections.

Host-serialization locks for per-host critical sections

When a service orchestrates N sibling child processes on a single host and some operations cannot safely run two-at-a-time on that host -- whether because of a framework constraint (Apple VZ save/restore) or because of shared-resource starvation (VZ teardown + WAL checkpoint + virtiofs drain all competing for main-thread and I/O bandwidth) -- park a tokio::sync::Mutex<()> on the service's shared state struct and acquire it at the top of the handler for the whole duration of the critical section. Mutex<()> isn't a weird construction: the unit value is the lock-token, the type signals "pure serialization, no protected payload". Semaphore::new(1) is equivalent -- pick one and stay consistent.

Current instances in crates/capsem-service/src/main.rs:

• save_restore_lock: serializes Apple VZ saveMachineStateToURL / restoreMachineStateFromURL across sibling VMs. Concurrent save/restore corrupts the VirtioFS ring state on the unlucky VM, surfaces as ext4-on-loop0 I/O errors after resume. Held through handle_suspend (IPC + child-exit wait) and handle_resume (spawn + wait_for_vm_ready). See docs/src/content/docs/gotchas/concurrent-suspend-resume.md.

• shutdown_lock: serializes VM teardown across handle_delete / handle_stop / handle_purge / handle_run. Without it, N concurrent deletes under load starve each other of the bandwidth each capsem-process needs to exit cleanly within the 1s fast-path budget; past the budget the service SIGKILLs mid-checkpoint and leaves a non-empty session.db-wal. Held through shutdown_vm_process for the whole SIGTERM + wait_for_process_exit window.

When to reach for this pattern:

• Symptom is "works solo, fails under concurrency on the same host."
• Root cause is a per-host resource, not per-VM: Apple VZ main thread, virtiofsd, DbWriter checkpoint, APFS fsync.
• Production runs exactly one service per host per user, so an in-process tokio mutex is enough -- no need for a file-lock or distributed primitive.

When NOT to reach for it:

• If the contention is per-VM (two handlers acting on the same VM), protect the VM entry in instances: Mutex<HashMap<...>> instead.
• If the "contention" is really a durability race (writer thread hasn't flushed), the right fix is usually the signal-handler explicit-cleanup pattern below, not another serialization lock.

Signal-driven explicit cleanup for background-thread owners

Any long-running Rust process that owns background threads (SQLite writer, notify PollWatcher, MCP aggregator subprocess, vsock relay) and runs under a bounded SIGTERM-to-SIGKILL budget must NOT rely on Drop + tokio-runtime-drop ordering to finish cleanup. On SIGTERM, hand owned resources to the signal handler and drain them synchronously BEFORE letting the main run loop return.

Symptom when this is missing: under concurrent teardowns on one host, the service SIGKILLs a child mid-checkpoint or mid-flush. Visible as session.db-wal left non-empty, missing fs_events rows, dangling aggregator subprocesses. Works solo, fails under -n 4.

Concrete primitives in this tree:

• DbWriter::shutdown_blocking(&self) — takes the stored mpsc sender, joins the writer thread, runs the final PRAGMA wal_checkpoint(TRUNCATE). Arc-safe: other Arc<DbWriter> clones remain valid but their writes become no-ops. Idempotent. Drop delegates to it.
• FsMonitor::shutdown_and_join(&self) — sends on the shutdown channel so the event loop runs its final flush, then joins the thread. Must run BEFORE DbWriter shutdown, because fs_events fan into DbWriter.
• CAPSEM_TEST_SLOW_CHECKPOINT_MS — test-only env var in writer_loop that inserts a sleep before the final checkpoint. Use in tests that need to distinguish explicit cleanup from implicit runtime-drop ordering.

Canonical wiring in crates/capsem-process/src/main.rs:

struct Shutdown {
    db: Option<Arc<DbWriter>>,
    fs_monitor: Option<FsMonitor>,
}

impl Shutdown {
    fn drain_blocking(&mut self) {
        // fs_events fan into DbWriter -- flush fs_monitor first.
        if let Some(m) = self.fs_monitor.take() { m.shutdown_and_join(); }
        if let Some(db) = self.db.take() { db.shutdown_blocking(); }
    }
}

// Populate as owners are constructed:
shutdown.lock().await.db = Some(Arc::clone(&db));
shutdown.lock().await.fs_monitor = Some(monitor);

// Signal handler drains through spawn_blocking, then stops the run loop:
rt.spawn(async move {
    /* wait on SIGTERM/SIGINT */
    let mut owned = std::mem::take(&mut *shutdown.lock().await);
    let _ = tokio::task::spawn_blocking( || owned.()).;
     { core_foundation_sys::runloop::(...); }
});

Key properties:

1. Deterministic order. The drain order is explicit (fs_monitor -> db), not "whatever reverse-declaration-order Drop happens to give us after tokio aborts tasks."
2. Synchronous join. The handler waits for each background thread to finish. No "hope the task finishes before the runtime drops."
3. Run loop stops last. CFRunLoopStop (macOS) fires only after drain returns. Main returns afterwards; the remaining tokio-runtime drop is now a no-op fast path because the heavy work already completed.
4. Arc-safe shutdown APIs. shutdown_blocking(&self) works through a shared Arc<DbWriter> — callers don't have to chase down every clone. Use std::sync::Mutex<Option<Sender>> internally; the hot-path write() clones the sender under the lock and releases it before .await.

When to reach for this pattern:

• The process has std::thread::spawn or tokio::task::spawn_blocking workers that run durability-critical work on shutdown (WAL checkpoint, queue flush, child-process wait).
• A parent sends SIGTERM then SIGKILLs after a short, fixed budget.
• Today's cleanup relies on Drop running inside tokio task abort — i.e., you can't draw a line between "cleanup finished" and "run loop exited."

Call out when NOT to use it:

• One-shot CLIs that exit on natural task completion (no run loop, no signal window).
• Workers whose only side effects are in-memory (no durability to lose).

When adding a new long-running process or a new background-thread owner, wire it through Shutdown from day one. Don't ship a new binary that "should be fine because Drop will run" — under load, Drop won't run in time.

Logging

• tracing crate with FmtSpan::CLOSE for timing spans
• RUST_LOG=capsem=debug for full boot timing breakdown
• RUST_LOG=capsem=info for top-level only
• Use structured fields: tracing::info!(domain = %domain, status = %code, "request completed")

One rule, one function

A rule that lives in its callers is a rule that will be wrong in some of them. Resolution, precedence, and lifetime rules belong in a function; callers call it.

This is the single most expensive pattern in this codebase's history. Two instances, both found the same day:

Rule	Copies	Outcome
"read the recent log"	4 (support_bundle, /service-logs, triage, service session tails)	rotation landed, one copy learned, two silently returned nothing -- a daemon logging errors reported none
"point Capsem at a temp root"	every fixture, by hand	CAPSEM_RUN_DIR/CAPSEM_ASSETS_DIR outrank CAPSEM_HOME, so setting the home alone read the caller's directories: green locally, broken in the gate

Both look locally correct in every copy. That is what makes them expensive: nothing is visibly wrong at any one site, and a fix applied to one never reaches the others.

Established wrappers -- use them, do not re-derive:

Need	Call	Never
Recent log content	telemetry::read_log_tail(stream, max)	File::open/fs::read on a *.log path
Enumerate a stream	telemetry::log_stream_files(stream)	read_dir + name filtering
Redirect paths in a fixture	paths::CapsemPathsGuard::redirect(root)	set_var("CAPSEM_HOME", ...)
Any ~/.capsem/... path	a paths:: helper	home.join(".capsem")
Checkpoint marker	paths::checkpoint_complete_path(cp)	rebuilding <name>.complete

tests/test_path_and_log_wrappers_are_mandatory.py enforces the first three.

When unifying, take the better implementation, not the first one. support_bundle's tail reader seeked to each file's end; the shared one read whole files. Guest console output is guest-controlled, so read-whole let a chatty VM decide how much memory capsem support allocated. The shared function now seeks.

Two copies that agree today are still a bug. checkpoint_complete_path existed in capsem-process and capsem-service, identical except that one hardcoded the fallback name and the other used a constant. Changing that constant would have left the process writing a resume marker the service never looked for, with nothing wrong at either site.

Lessons learned

1. Content-Encoding: Always handle response decompression generically. Gzip compressed SSE responses caused NULL telemetry because the parser got binary garbage. Never strip Accept-Encoding as a workaround.

2. Platform type widths: as _ is your friend for cross-platform libc calls. Explicit casts (as c_ulong) will fail on the other platform.

3. Debouncer timing: If a VM shuts down before debounced events flush, telemetry is lost. Add sleep 1 in test commands, or use explicit flush on shutdown.

4. VirtioFS whiteouts: Apple VZ's VirtioFS doesn't support mknod, so overlayfs can't use it directly as upper. The ext4 loopback workaround provides full POSIX.

5. setsid for controlling terminal: Without setsid, the PTY has no foreground process group and Ctrl-C (SIGINT) is not delivered. capsem-init uses setsid to fix this.

6. serde_json::Value on LLM hot path: Three ai_traffic struct fields (ResponseInfo.output, FunctionResponse.response, FunctionCall.args) used serde_json::Value for large payloads that were only stringified. This forced full DOM allocation on every streaming request. Fixed by removing unused fields and switching to Box<serde_json::value::RawValue>.

7. Prefer syscalls over subprocesses: std::process::Command costs 5-30ms per spawn (fork/exec). If a syscall does the same thing, use it. Example: cp -c -R for APFS clonefile was 20-30ms; direct libc::clonefile() is <1ms. On Linux, ReflinkSnapshot already uses FICLONE ioctl directly -- no subprocess. Always check if the OS provides a syscall before reaching for Command.

8. Blocking I/O in MCP file tools: All 7 snapshot file tool handlers ran blocking I/O (clonefile subprocess, walkdir, blake3) directly on tokio worker threads while holding a tokio::sync::Mutex. The auto-snapshot timer did the same. This caused snapshot creation to hang from the model's perspective. Fixed by wrapping in spawn_blocking everywhere.

9. Single-file CoW: Added helper that uses APFS clonefile on macOS and FICLONE on Linux for instant CoW copies. Used in snapshot compact (host-to-host). (snapshot-to-VirtioFS-workspace) because APFS clonefile is metadata-only and VirtioFS may serve stale data to the guest. Revert must use (byte copy) so the guest sees the new content immediately.

Async reference

Read references/rust-async-patterns.md for comprehensive tokio patterns (tasks, channels, streams, error handling). From the community (6.4K installs).