debugging-computerd-fuse

# Debugging computerd against real FUSE This skill captures the recipe for isolating bugs in computerd that only fire when the kernel-FUSE backend is the one wired up. Most unit tests run against `SQLiteTestStorage` and an in-process VFS or the userspace `FUSE_MOUNT=shim` polling driver — neither exercises the kernel-FUSE callback path. Several production-only bugs (write buffer never spilled to the VFS, `spawn(cwd=/mount)` deadlocks) have shipped to `next` and not been caught by the regular suites because the harness is hard to drive end-to-end from a dev box. The workflow here bypasses workerd, wrangler, and the vitest-pool-workers harness. It boots the computerd SEA binary in a real docker container with `/dev/fuse` and SYS_ADMIN, and drives its capnweb endpoints from a plain Node script. ## When to reach for this - A user report of "wrote a file via FUSE, RPC reads see 0 bytes." - A report of "exec hangs forever" or "computerd `/health` stops responding after the first shell call." - Adding a new computerd op (FUSE callback, sync RPC, runner feature) and you want a smoke test that doesn't depend on Docker-in-DO or Cloudflare Containers. - The harness vitest suite (`packages/computer/src/test-harness/`) hangs at `ws.shell.exec` against the docker container and you need to attribute the wedge to computerd vs workerd vs the network shim. - Confirming a fix actually settles in the deployed binary: the SEA build is what wrangler ships, and reproducing locally with the same binary is the only way to be sure. ## What you need - Docker, with `/dev/fuse` and `--privileged` available. In Pi's sandbox this works through DinD; on a Linux dev box it works natively; on macOS you need `colima start --vm-type=vz` (the default qemu vm doesn't surface `/dev/fuse`). - A fresh computerd binary at `artifacts/computerd/computerd-linux-x64`. Build it with `npm run build:bin --workspace @cloudflare/computerd`, or just `npm run build:all` from the repo root if you need the docker image too. ## Boot a real-FUSE computerd container There's already a recipe — `packages/computer/test-harness/run-computerd.sh`. It picks a host port, runs the binary with `--privileged --device /dev/fuse --cap-add SYS_ADMIN --cap-add MKNOD` on Debian trixie with fuse3 from apt (computerd links the system libfuse 3 and needs 3.17 or newer), waits for `/health`, prints the URL on stdout and the container id on stderr. It forwards `MOUNT_IGNORE`, `MOUNT_IGNORE_PATH`, and any `COMPUTERD_FUSE_*` variable from your environment into the container. ```bash # Start COMPUTERD_HARNESS_PORT=18080 bash packages/computer/test-harness/run-computerd.sh # prints: http://127.0.0.1:18080 on stdout # prints: COMPUTERD_HARNESS_CID=<id> on stderr # Verify curl -sf http://127.0.0.1:18080/health docker logs <CID> | tail # should see: computerd listening on 0.0.0.0:8080 mount=/workspace backend=fuse # Tear down docker kill <CID> ``` In a tmux session: ```bash tmux new-session -d -s computerd \ "COMPUTERD_HARNESS_PORT=18080 bash packages/computer/test-harness/run-computerd.sh; sleep 600" for i in 1 2 3 4 5; do sleep 5 if curl -sf http://127.0.0.1:18080/health 2>/dev/null; then echo ready; break; fi done ``` **`backend=fuse` is the success signal.** If you see `backend=shim` the FUSE mount didn't take and you're testing the wrong path. `backend=none` means `FUSE_MOUNT=none` was set; `FUSE_MOUNT=fuse` with a missing `/dev/fuse` would have failed startup outright. ## Drive computerd from a Node script computerd serves a composite `WorkspaceRPC` over `/api`, a capnweb WebSocket. That is the only RPC carrier: capnweb's HTTP batch transport cannot deliver a returned stream, which is what every read on this interface is. The `@cloudflare/computer-rpc/client` package wraps the socket and the `/driver` subpath exposes `pushOnce`/`pullOnce` against a Node-side `Database`. Set up a probe project once: ```bash mkdir -p /tmp/computerd-probe && cd /tmp/computerd-probe cat > package.json <<'EOF' { "type": "module", "private": true, "dependencies": { "@cloudflare/dofs": "file:/workspace/packages/dofs", "@cloudflare/computer-rpc": "file:/workspace/packages/rpc", "ws": "^8.18.0" } } EOF npm install --ignore-scripts --no-audit --no-fund ``` Now you can write probes that hold both sides of the wire. The canonical shape, with the WebSocket impl pinned to the `ws` package (Node's built-in `WebSocket` doesn't negotiate the permessage-deflate extension that computerd advertises): ```js import { Database, WorkspaceFilesystem, initializeSchema } from "@cloudflare/dofs"; import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; import { createWorkspaceClient } from "@cloudflare/computer-rpc/client"; import { pullOnce, pushOnce } from "@cloudflare/computer-rpc/driver"; import { WebSocket } from "ws"; const url = process.env.COMPUTERD_URL; // e.g. http://127.0.0.1:18080 const wsUrl = `${url.replace(/^http(s?):\/\//, "ws$1://")}/api`; const storage = new SQLiteTestStorage(); const db = new Database(storage); initializeSchema(db, () => Date.now()); const fs = new WorkspaceFilesystem(db, { now: () => Date.now() }); // Stage some local state. IMPORTANT: paths must be inside the // container's MOUNT_POINT (default /workspace) or computerd's FUSE // driver won't surface them on the host — computerd serves only the // `/workspace/...` subtree of vfs_nodes through the kernel mount. await fs.mkdir("/workspace", { recursive: true }); await fs.writeFile("/workspace/probe.txt", "hello from probe"); const client = createWorkspaceClient({ url: wsUrl, WebSocketImpl: WebSocket }); try { const pushed = await pushOnce(db, client.sync); console.log("pushed", pushed); const result = await pullOnce(db, client.sync); console.log("pulled", result); } finally { await client.close(); } ``` Run with `COMPUTERD_URL=http://127.0.0.1:18080 node probe.mjs`. To verify the push actually landed on the FUSE side: ```bash docker exec <CID> sh -c 'ls -la /workspace && cat /workspace/probe.txt' ``` If `ls /workspace` is empty but the push reported `pushed=N`, the RPC apply path landed bytes in `vfs_nodes` but FUSE isn't surfacing them. Check that: - Your absolute paths start with `MOUNT_POINT` (default `/workspace`). Writing to `/probe.txt` succeeds at the RPC layer but FUSE only serves the `MOUNT_POINT` subtree. - computerd actually logs `backend=fuse` (not `backend=shim`). - You aren't reading the wrong container — `docker ps` after a zombie `docker rm -f` can show stale entries. ## Drive container-side writes The cleanest way to simulate a container-side FUSE write — the exact path computerd was designed to handle — is `docker exec` into the running container: ```bash docker exec <CID> sh -c 'echo container-write > /workspace/r2/hello.txt' ``` This is a real FUSE write: kernel → fuse-napi → computerd's `write` op → computerd's in-memory buffer for that file → on `release`/`flush`/`fsync` the buffer spills into the backing VFS (commit `68407fc`). On the host side you can then `pullOnce` and see the new entry land in `applyChanges`. Use `docker exec` when you need to isolate the FUSE mount from computerd's command runner: computerd is not the parent of that process. `client.shell.exec()` also works on current binaries; it exercises the command runner and its removal guard as well. ## Known deadlock patterns ### FUSE buffer never spilled to the VFS Symptom: container `cat /workspace/foo` returns the bytes you wrote, but a host-side `pullOnce` sees the file at size 0. Mechanism: the FUSE driver in `packages/computerd/src/fuse/driver.ts` buffers writes in an in-memory `files` Map keyed by path. Reads through FUSE pull from that buffer, but the backing VFS only sees the empty inode the `create` op registered. If `release` / `flush` / `fsync` don't spill the buffer, RPC consumers see an empty file. Fix shipped: commit `68407fc` — `flushEntry(path)` runs on each of those ops. Test for it: `packages/computerd/src/fuse/driver.test.ts` and the companion test at `8111d68`. Mirror that shape if you find a new buffer-vs-VFS desync. ### computerd's event loop blocked in spawn Historical failure: an older `Runner.exec` implementation made `client.shell.exec` hang under real FUSE. `/health` stopped responding; `docker exec <CID> ls /workspace` then hung too. Mechanism: it called `spawn("/bin/sh", { cwd: "/workspace" })`. libuv's `uv_spawn` forks and waits on a status pipe for the child to exec. The child, between fork and exec, does `chdir("/workspace")`. The kernel issues a FUSE `LOOKUP` against the computerd mount. The callback needs to run on computerd's event loop, which is blocked in the pipe read. Deadlock. Diagnostic: ```js // Against a fresh computerd, both calls should return. await client.shell.exec({ command: "echo hi", cwd: "/tmp", timeoutMs: 5000 }); await client.shell.exec({ command: "echo hi", cwd: "/workspace", timeoutMs: 5000 }); ``` Same computerd, same client, same network — only `cwd` differs. If the second hangs, the old deadlock has regressed. The runner now changes directory inside the **shell**, not in `spawn`'s fork-and-exec path. It prefixes the command with `cd ... &&` so FUSE lookups run after the shell starts and computerd's event loop is responsive. It checks cwd existence through the VFS before spawning to preserve the ENOENT error contract. ### Other event-loop blockers to suspect Any computerd code path that does sync work against `MOUNT_POINT`/the FUSE mount can wedge: `fs.statSync(cwd)`, `fs.readdirSync(MOUNT_POINT)`, `fs.realpathSync(...)` — all synchronous, all block the event loop, all issue FUSE callbacks to computerd itself. Rule of thumb when adding computerd code: anything that goes through Node's `fs` against the mount point must be async (so libuv's threadpool services the call) OR must read the dofs `Database` / VFS directly (pure SQL, no FUSE involvement). ## Recipe library ### Is passthrough engaged? With `MOUNT_IGNORE` set, an open of a local-only file should hand the file to the kernel and keep computerd out of the data path. Two ways to check, from inside the container: ```bash # What the mount negotiated, and how opens were served. curl -s localhost:8080/__computerd/info | jq .ignore.fastPaths # Whether data still reaches the daemon. Boot with # COMPUTERD_FUSE_TRACE=summary and COMPUTERD_FUSE_TRACE_FILE=/tmp/trace.json, # do the I/O, then dump the counters. read and write should not move # for local-only files. kill -USR2 1 && jq '.ops[] | select(.op == "read" or .op == "write")' /tmp/trace.json ``` Keep the trace file outside the mount: a process serving a FUSE mount deadlocks if it writes into that mount. `passthroughReason` names the likely cause when passthrough is off, and `COMPUTERD_FUSE_PASSTHROUGH=0` turns it off on purpose, to rule it in or out of a bug. ### Check native roots With `MOUNT_IGNORE` configured, eligible local-only directories can be mounted from the backing directory onto `/workspace`. Check the live state and the kernel's mount table separately: ```bash curl -s localhost:8080/__computerd/info | jq .ignore.fastPaths.nativeRoots docker exec <CID> sh -c "grep ' /workspace/' /proc/self/mountinfo" ``` The info response shows `active`, `failures`, and `reason`; stats also include `lastFailure`. A refused bind remains available through FUSE. A symlink in either source or target path is refused by the native helper, not followed by a shell mount command. Disable native roots with `COMPUTERD_FUSE_NATIVE_ROOTS=0` when isolating a problem. To remove a root before `rm -rf`, call the daemon's detach endpoint: ```bash curl -X POST localhost:8080/__computerd/native-roots/detach \ -H 'content-type: application/json' \ -d '{"path":"node_modules"}' ``` If `RPC_CLIENT_SECRET` is set, include `Authorization: Bearer <secret>`. Do not use a pathname-based `umount` on a workspace path that might be a symlink. On shutdown and before mounting FUSE at startup, computerd lazily detaches everything it finds below the workspace mount; do not put unrelated mounts there. ### Reset between probes A wedged computerd container can leave FUSE zombie processes that `docker kill` refuses to reap. The cleanest fix: ```bash # Try kill first docker kill <CID> 2>&1 # If it complains about a PID being a zombie, just abandon that # port and pick a new one COMPUTERD_HARNESS_PORT=18081 bash packages/computer/test-harness/run-computerd.sh ``` The zombie containers don't hold the host port once docker has nominally released them, but they linger in `docker ps` until the next daemon restart. Harmless for further work. ### Stage a read-only mount fixture Mirrors what the workspace indexer does. Useful when reproducing M3.5-class issues (read-only mount enforcement) without going through the full Workspace constructor: ```js import { Database, ROOT_INODE, WorkspaceFilesystem, initializeSchema, invalidateReadOnlyMountCache } from "@cloudflare/dofs"; import { SQLiteTestStorage } from "@cloudflare/dofs/testing"; const storage = new SQLiteTestStorage(); const db = new Database(storage); initializeSchema(db, () => Date.now()); const fs = new WorkspaceFilesystem(db); const root = "/workspace/r2"; // Indexer step 1: row at read-write so writeFile can land db.run("INSERT INTO _vfs_mounts (root, kind, mode, indexed) VALUES (?, ?, 'read-write', 0)", root, "fixture"); invalidateReadOnlyMountCache(db); // Indexer step 2: materialize await fs.mkdir(root, { recursive: true }); await fs.writeFile(`${root}/hello.txt`, "hello world"); // Indexer step 3: stamp mount_root on every inode under the root. // Path-to-inode resolver: function resolveInode(db, absPath) { const parts = absPath.split("/").filter(Boolean); let inode = ROOT_INODE; for (const part of parts) { const row = db.one("SELECT child_inode FROM vfs_dirents WHERE parent_inode = ? AND name = ?", inode, part); if (!row) return undefined; inode = row.child_inode; } return inode; } const rootInode = resolveInode(db, root); const queue = [rootInode]; const subtree = [rootInode]; while (queue.length) { const parent = queue.shift(); for (const c of db.all("SELECT child_inode FROM vfs_dirents WHERE parent_inode = ?", parent)) { subtree.push(c.child_inode); queue.push(c.child_inode); } } for (const inode of subtree) { db.run("UPDATE vfs_nodes SET mount_root = ? WHERE inode = ?", root, inode); } // Indexer step 4: flip to read-only and invalidate the cache db.run("UPDATE _vfs_mounts SET mode = 'read-only', indexed = 1 WHERE root = ?", root); invalidateReadOnlyMountCache(db); ``` After staging this and `pushOnce`'ing to computerd, the container's FUSE view will surface `/workspace/r2/hello.txt`. Container-side writes to that subtree will propagate back via the next `pullOnce`, where `applyChanges` skips them with `reason: "read-only"`. ### Reference: a full M3.5-style probe For the FUSE write/pull path, use `script/computerd-fuse-flush.mjs` as the starting point for a probe that stages a mount and checks the applied/skipped split.