# rch — Remote Compilation Helper
<div align="center">
<img src="rch_illustration.webp" alt="rch - Remote Compilation Helper for AI coding agents">
</div>
<div align="center">
<h3>Quick Install</h3>
```bash
curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/remote_compilation_helper/main/install.sh?$(date +%s)" | bash -s -- --easy-mode
```
<p><em>Installs `rch` + `rchd`, bootstraps config, and can install/start the background daemon. If remote execution cannot proceed, RCH fails open to local execution.</em></p>
</div>
<div align="center">
<img src="rch_diagram.webp" alt="rch architecture diagram">
</div>
<div align="center">
**Transparent remote compilation for multi-agent development**
[](LICENSE)
[](https://www.rust-lang.org/)
[](https://codecov.io/gh/Dicklesworthstone/remote_compilation_helper)
</div>
---
## TL;DR
**Problem**: Many concurrent AI agents can saturate local CPU and make your workstation unusable.
**Solution**: RCH runs as a Claude Code PreToolUse hook, classifies build-like commands in milliseconds, executes them on remote workers, and returns artifacts/output as if they ran locally.
**Design constraint**: RCH is fail-open. If remote execution is not safe/possible, commands run locally.
---
## What RCH Intercepts
RCH currently recognizes and can offload:
| Ecosystem | Intercepted Commands |
|---|---|
| Rust | `cargo build`, `cargo check`, `cargo clippy`, `cargo doc`, `cargo test`, `cargo nextest run`, `cargo bench`, `rustc` |
| Bun/TypeScript | `bun test`, `bun typecheck` |
| TypeScript | `tsc --noEmit`, `npx tsc --noEmit` |
| Go | `go build`, `go build -o <file>`, ordinary `go test`, `go test -c -o <file>`, `go vet` |
| C/C++ | `gcc`, `g++`, `clang`, `clang++` |
| Build Systems | `make`, `cmake --build`, `ninja`, `meson compile` |
| Nix | `nix build`, `nix-build`, `nix flake check`, `nix develop -c <cmd>`, `nix shell -c <cmd>` |
Nix builds only route to workers that advertise a `nix` capability (a usable `nix`
binary plus a populated `/nix/store`); on a fleet with no such worker they fall
back to local execution (or are refused under `RCH_REQUIRE_REMOTE=1`, exactly as
with Bun/Node). Nix outputs stay in the worker's `/nix/store` behind a `result`
symlink, so these run as streaming, exit-status-only commands (no artifacts are
copied back — the flake source is synced out, the build runs, the result stays
remote).
Go builds support an explicit, project-relative **file** output, for example
`go build -o bin/app ./cmd/app` or `go build -o 'products/app [dev]*?' main.go`.
The output is returned to that exact path, including when `CARGO_TARGET_DIR` is
set. Common build flags such as `-p`, `-tags`, `-trimpath`, and `-race` are
supported, as are linker stripping flags (`-ldflags '-s -w'`) and `-X` variable
definitions. Builds require the durable Unix execution path and a local Go
installation. Both endpoints must report empty effective `GOFLAGS` and the
dispatcher's native `GOOS`/`GOARCH`. The caller's selected Go version, CGO
configuration, experiments, and architecture tuning settings are captured
before upload and must match the worker exactly before compilation starts.
Ordinary builds with CGO enabled remain supported when those settings match.
The worker compiles into a fresh private file outside the source root, preserving
wildcard `go:embed` inputs. Only after successful compilation produces a regular
file does it create output parents and an adjacent stage for atomic replacement
of the previous worker file. Retrieval then stages and validates that file before
replacing the local output. A missing download cannot be satisfied by an old
local binary, and durable recovery resumes collection without rerunning Go.
Directory outputs, symlinks or symlinked output parents, cross compilation,
native-library modes such as `c-shared`/`c-archive`, and low-level flags that can
create extra files are outside this output contract. Implicit forms such as
`go build`, `go build .`, and `go build ./...` use Go's own package query to
prove that exactly one main package is selected and bind its executable name
before execution.
Compile-only tests also return an exact binary: for example,
`go test -c -o bin/package.test ./package`. Both endpoints verify that the same
single package has test sources; packages without tests and selections spanning
multiple packages stay local. Compilation never runs the test binary. The binary
uses the same fresh-file publication and collection-only recovery as `go build`,
and its caches and temporary files stay outside the selected source tree.
Package discovery for implicit builds and compile-only tests requires Go 1.18
or newer with workspace mode disabled. Test profiling, fuzz corpora, implicit
test-binary names, and output-producing test execution (`-coverprofile`,
`-cpuprofile`, `-trace`, `-fuzz`, `-o` without `-c`, and related `-test.*` forms)
stay local. Strict remote mode refuses unsupported forms instead of executing
them locally.
RCH explicitly does **not** intercept local-mutating or interactive patterns (examples):
- Package management: `cargo install`, `cargo clean`, `bun install`, `bun add`, `bun remove`
- Bun runners/dev: `bun run`, `bun build`, `bun dev`, `bun x` / `bunx`
- Nix interactive/mutating: bare `nix develop` / `nix shell`, `nix run`, `nix repl`,
`nix profile`, `nix flake update`, `nix store gc`, `nix-env`, `nix-shell`
- Watch/background/piped/redirected commands where deterministic offload is unsafe
### Yarn Berry / Plug'n'Play preparation
Linux workers can prepare Yarn Berry projects using the standard project-local
PnP layout. Set `enableGlobalCache: false` in `.yarnrc.yml`, keep the default
cache, unplugged, virtual and install-state paths, and declare TypeScript as a
dependency of the workspace that runs the check. The worker's Yarn version
must match an authored `packageManager: "yarn@VERSION"` declaration.
Berry configuration probes require Linux descendant supervision, including
when Berry uses the `node-modules` linker; Yarn Classic keeps its existing
preparation path.
```bash
rch exec -- tsc --noEmit --incremental false --composite false
```
This PnP launch accepts literal bare `tsc` arguments with explicit non-emitting,
non-incremental, non-composite settings. Disabling both incremental and
composite output prevents inherited `tsconfig.json` settings from writing
build-info files, including when the compiler exits with an error. It uses
Yarn's dependency binary resolver, so a
package script or global `tsc` cannot replace the prepared compiler. Shell
wrappers, `npx`, Bun, watch/build modes, and emitting overrides are outside this
PnP launch contract.
Qualified workspace checks run from the workspace root or a selected member's
package directory. Member-local Yarn configuration and environment-file
overrides that would change resolution from the prepared root are refused.
Live `portal:` / `link:` dependencies and `pnpIgnorePatterns` are also outside
this initial contract: dependency resolution must stay within the prepared
package graph.
Preparation runs an immutable install. Source-qualified clean-overlay runs
bind workspace preparation to the selected workspace root, record exact
generated loaders, local cache/unplugged files and installation state, and
rehash them before skipping a later install. Ordinary preparation reruns
Yarn's cached immutable install. Committed zero-install loaders and cache files
remain source; `.yarn/releases`, plugins and patches are never admitted as
generated dependency outputs. Global or custom PnP output paths require a
separate ownership contract and are refused.
---
## Why It Works Well
- Transparent hook behavior: agents see normal command semantics.
- 5-tier classification pipeline optimized for very fast non-compilation rejection.
- Daemon-owned worker state and slot accounting.
- Cache-aware worker selection and project affinity.
- Queue + cancellation primitives for overloaded scenarios.
- Deterministic reliability subsystems for convergence, pressure handling, and remediation.
- Unified status surface with posture and remediation hints.
---
## Current Architecture
```text
Agent Shell / Claude Code
|
v
PreToolUse Hook -> rch (classifier + hook protocol)
|
v
rchd (daemon)
- worker selection
- queueing and cancellation metadata
- health, alerts, telemetry, history
- reliability subsystems (convergence, pressure, triage)
|
v
Remote workers (rch-wkr)
- execute build/test commands
- manage worker cache
- report capabilities/health/telemetry
```
Workspace crates:
- `rch/`: Hook + primary CLI
- `rchd/`: Local daemon + scheduling/reliability APIs
- `rch-wkr/`: Worker execution/caching agent
- `rch-common/`: Shared protocol/types/patterns/UI foundations
- `rch-telemetry/`: Telemetry collection/storage integration
---
## Reliability Model (Operational)
RCH now includes a deterministic reliability stack for multi-repo and multi-worker stability:
- **Path-dependency closure planning**: builds can include required repository closure rather than a single root.
- **Canonical topology enforcement**: worker/project roots are normalized around `/data/projects` and `/dp` conventions.
- **Repo convergence service**: tracks worker drift vs required repos and can repair drift.
- **Disk pressure resilience**: pressure scoring, admission control, safe reclaim with active-build protection.
- **Process triage/remediation**: bounded TERM/KILL escalation with audit trail.
- **Cancellation orchestration**: deterministic cancellation metadata and worker health integration.
- **Unified posture/reporting**: status output includes posture, convergence state, pressure, and actionable remediation hints.
---
## Installation
### Recommended: Installer
```bash
curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/remote_compilation_helper/main/install.sh?$(date +%s)" | bash -s -- --easy-mode
```
The installer also refreshes the `rch` agent skill in `~/.claude/skills/rch`
and `~/.codex/skills/rch`. It leaves both directories untouched when `jsm`
reports that it manages an installed `rch` skill, or when you pass
`--skip-skill` (or set `RCH_SKIP_SKILL=1`, which suits unattended updaters).
Skipping the skill never affects the binaries, the hook or the daemon.
### From Source
```bash
git clone https://github.com/Dicklesworthstone/remote_compilation_helper.git
cd remote_compilation_helper
cargo build --release
cp target/release/rch ~/.local/bin/
cp target/release/rchd ~/.local/bin/
```
### Source Build Note
All dependencies — including the FrankenTUI (`ftui-*`), `rich_rust`, and TOON
(`tru`) crates — resolve from crates.io, so a clean `git clone && cargo build`
builds on any machine with no special directory layout or pre-cloned
dependency tree required.
### rsync Requirement (macOS note)
RCH moves source and artifacts with `rsync` over SSH and works best with
**rsync 3.2 or newer** (`--info=progress2` progress, zstd compression,
`--append-verify` resumes). On Linux the distro rsync qualifies.
On **macOS**, the stock `/usr/bin/rsync` is *openrsync* (macOS 15+) or Apple's
rsync 2.6.9 (older releases); neither accepts rsync 3.x flags. RCH handles this
automatically: it probes `rsync --version`, prefers a Homebrew/MacPorts rsync
(`/opt/homebrew/bin/rsync`, `/usr/local/bin/rsync`, `/opt/local/bin/rsync`)
even when it is not first on `PATH`, and otherwise drives the stock binary with
a compatible flag set (`--progress --stats -vv`, zlib instead of zstd, and the
zero-build-output detector fails open). For full-speed transfers:
```bash
brew install rsync
```
`rch doctor` reports which binary and flavour RCH resolved. To pin one
explicitly, set `[transfer] rsync_bin = "/path/to/rsync"` or export
`RCH_RSYNC_BIN`.
---
## First-Time Setup
### Fastest Path
```bash
rch init
```
`rch init` can guide:
1. Worker discovery from SSH config/aliases
2. Worker probing and selection
3. `rch-wkr` deployment
4. Toolchain synchronization
5. Daemon startup
6. Hook installation
7. Validation build
### Manual Path
```bash
# 1) configure workers
mkdir -p ~/.config/rch
cat > ~/.config/rch/workers.toml << 'TOML'
[[workers]]
id = "css"
host = "203.0.113.20"
user = "ubuntu"
identity_file = "~/.ssh/id_rsa"
total_slots = 32
priority = 100
TOML
# 2) start daemon
rch daemon start
# 3) verify workers
rch workers probe --all
# 4) install hook
rch hook install
# 5) check posture
rch check
rch status --workers --jobs
```
---
## Command Surface
Global flags:
```bash
-v, --verbose
-q, --quiet
-j, --json
-F, --format json|toon
--color auto|always|never
--no-color
--schema
--help-json
--robot-triage
```
### Core Operations
```bash
rch daemon start|stop|restart|status|logs|reload
rch workers list|capabilities|probe|benchmark|compare|drain|enable|disable
rch workers init|discover|setup|deploy-binary|sync-toolchain
rch status [--workers] [--jobs]
rch check
rch queue [--watch|--follow]
rch cancel <id> | --all
rch gc [--root DIR]... [--workers <id>...] # PREVIEW stale rch runtime dirs
rch gc --apply [--root DIR]... # ...and actually collect them
rch cache warm|clean|status [--workers <id>...] # remote source/target caches
rch rabs gc plan|run|history [--cas-root DIR] [--mode normal|emergency]
rch rabs worker|doctor|inventory
rch rabs worker reconcile <WORKER>
```
On macOS, daemon autostarts defer to a registered `com.rch.daemon` launchd
service without restarting its running process. The managed daemon waits if
another daemon still owns its socket. A private daemon bypasses this service
discovery when given both a nondefault socket and an explicit workers
configuration; a custom socket alone cannot be delegated to the shared service.
Fleet GC reports progress on stderr as workers finish. Each worker has a
15-minute budget shared across connections, scanned roots, and collection
batches; set `rch gc --worker-timeout=120` for a shorter budget. A timeout makes
the command exit nonzero while preserving completed workers' results. JSON/TOON
output contains one final response with those results and the error.
On timeout, GC terminates its foreground SSH process and allows two additional
seconds to reap it; cleanup failures are reported. This does not confirm whether
a remote collection batch finished: an interrupted `--apply` batch is reported
as having an unknown outcome, alongside confirmed earlier batches. The default
invocation remains a preview.
`rch cache status` also scans workers concurrently, with one 15-minute deadline
and a 10 MiB output budget per worker across all roots. Set
`rch cache status --worker-timeout=120` to shorten the deadline. Completed
workers and roots remain in the report if another scan fails; a timeout exits
nonzero, with one final JSON/TOON error response retaining those results.
Windows cache status and GC inspect the same `C:/rch` base as the tar transport,
plus the worker's resolved Cargo cache root. Every Windows directory remains
kept until native process/handle ownership checks are available; `--apply`
reports an explicit refusal. See [Windows cache inventory](docs/runbooks/worker-recovery.md#inspecting-windows-build-and-cargo-caches)
for additional roots and incomplete-scan behavior.
Capability refreshes skip duplicate requests for a worker while its probe is
running. Workers reuse verified Rustup inventory across requests and invalidate
entries when installed toolchains or component metadata change. Inventory runs
in bounded batches; unfinished or failed entries appear in `probe_warnings`,
and their component capabilities remain unavailable until verified. Disk, load,
and project-topology observations are collected again for each request.
### Hook + Agent Integration
```bash
rch hook install|uninstall|status|test
rch shim install|status|uninstall # cargo shim: offload builds started by scripts/Makefiles, not just hooked tool calls
rch agents list|status|install-hook|uninstall-hook
rch diagnose "cargo build --release"
rch admit "cargo build --release" # read-only preflight: offload / local / queue / defer verdict
rch admit --job --require-tool clang -- ./run_shards.sh # same, for a job-mode admission
rch why miss|refusal # RABS: explain a cache-key miss diff or an index refusal code
rch exec -- cargo build --release
rch --robot-triage --json
rch capabilities --json
rch robot-docs guide
```
RABS offers an experimental, opt-in worker-local dependency cache through
`cargo rabs --rabs-remote check`. See the [setup and measured validation
boundaries](docs/rabs-rch-worker-cache.md) before enabling it. Correct cache
serving has passed real Cargo/SSH tests. Four native Linux scenarios also passed
real compiler execution, replay, delivered binaries, and persistent worker/SDK
reuse over authenticated loopback TLS. Practical acceleration remains under
qualification; the measured whole-project warm run was slower than ordinary RCH.
A configured shared canonical coordinator is also available through
`cargo rabs --rabs-canonical-remote PROFILE.json -- build --workspace`.
This explicit route uploads captured inputs over pinned mutual TLS, runs Cargo
at the coordinator, shares eligible dependency compilations across clients and
returns verified build artifacts to a fresh private destination. Reusable
profiles support Rustup toolchain selection and a separate output root. See
[remote Cargo profiles and command support](docs/rabs-canonical-cargo.md#submit-through-cargo-with-a-reusable-remote-profile)
for setup, supported commands and the current qualification limits.
### Named Cargo JSON for installer consumers
An installer selecting explicit binary targets can request caller-visible
Cargo JSON through the canonical Cargo shim:
```bash
RCH_CARGO_JSON_STDOUT=1 RCH_REQUIRE_REMOTE=1 cargo build --locked --release \
--bin my-tool --bin my-alias --message-format=json,json-render-diagnostics
```
This opt-in requires the durable POSIX worker route and refuses local fallback,
arbitrary jobs, machine envelopes, and unsupported target selections. Selected
binary records reach stdout after artifact publication and acknowledged worker
release. Their executable, filenames, manifest, and source paths refer to the
caller tree. Selected output files are copied into an invocation-private retained
directory beneath the caller's actual target root and checked against the exact
publication fingerprints. A later build cannot replace these copies by publishing
into the shared target paths. Each record carries an `rch` extension retaining the original worker
record, exact retained stdout receipt, wrapper/build identity, and executable
BLAKE3 publication fingerprint. Unselected dependency records keep their worker
metadata and do not assert local delivery of intermediates. Diagnostics stay on
stderr. Compiler failure stdout comes from the same invocation's immutable
nonzero completion receipt; a successful compile with failed artifact delivery
advertises no caller-local success records. The ordinary hook and `rch --json
exec` envelope contracts are unchanged.
The delivery JSON, its private file paths, and their fingerprints are persisted
before source retirement and worker release. After the original wrapper has
exited, recover that same invocation without compiling again:
```bash
rch jobs recover ORIGINAL_WRAPPER_ID --cargo-json
```
This explicit output mode requires an invocation originally admitted with
`RCH_CARGO_JSON_STDOUT=1`; it cannot upgrade an ordinary job. It returns verified
retained stdout only after the original recovery and acknowledgement complete,
and uses the original exit status. Repeating it returns the first delivery's
exact JSON bytes and private paths, including after later builds or a failed
stdout write. A live unacknowledged wrapper or unknown owner identity refuses
detached recovery. `--json` machine envelopes conflict with `--cargo-json`.
The complete compiler stderr is collected from the same supervisor log with
the original completion status and retained before retirement. Its path,
length, BLAKE3 and compiler status appear in the selected record's receipt.
Recovery verifies those bytes and writes them to stderr without line or UTF-8
conversion. Failed compiler stdout and stderr use the exact nonzero completion;
no successful Cargo record is advertised when artifact delivery failed. Older
completed journals lacking retained diagnostics refuse this explicit output
mode rather than fabricate diagnostic parity.
Use the exit status together with the record binding. This route does not
provide native Windows delivery, an installer transaction, a source-content
receipt, or performance qualification. `scripts/e2e_cargo_json_stdout.py` checks
the actual shim, retained producer bytes, both real native binaries, cached
delivery, completed same-id stdout recovery, a broken stdout pipe, and a genuine
Cargo target-selection failure against an externally
admitted source and artifact root. No test result is implied by this description.
Wrapper-loss, pending recovery, cancellation, full installers, native Windows,
and same-invocation live-incumbent/A/A performance acceptance remain open.
### Job Mode (non-compilation workloads)
`rch exec --job` admits an arbitrary NON-compilation workload (sharded tests,
fuzzing, benchmarks, mutation testing) onto the same remote rails. It bypasses
ONLY the compilation classifier — the PreToolUse hook never sets it, so
auto-delegation of ordinary commands is impossible.
```bash
rch exec --job -- ./run_shards.sh
# Declared result directories sync back on ANY exit code (including failures):
rch exec --job --result-dir fuzz/corpus --result-dir crashes -- ./fuzz_target.sh
```
Semantics: the remote exit status surfaces verbatim; no toolchain/worker-env
rerun heuristics apply. A declared `--result-dir` that is missing or only
partially transferable fails loudly (`RCH-E309`, exit 102) regardless of the
job's own exit status. Paths must be repository-relative; conflicts with
`--clean-overlay` / `--source-content-receipt` are refused.
Native Apple commands can use job mode too: direct `xcodebuild` and `xcrun`
invocations require a worker declaring `os = "darwin"`. This preserves the OS
fence: ordinary jobs cannot consume that worker, and Linux workers cannot take
an Xcode job. Invoke the tool directly; RCH does not inspect arbitrary scripts
or infer a Mac requirement from text merely mentioning Xcode.
```bash
RCH_REQUIRE_REMOTE=1 RCH_BUILD_SLOTS=1 rch exec --job -- \
xcodebuild -project ios/App.xcodeproj -scheme App -jobs 1 build
```
This is host routing, not proof that Xcode, signing credentials, or a Simulator
are ready. Native test and Simulator safety checks still apply. Both `rch` and
`rchd` need the native-command routing support; keep explicit strict-remote
mode when qualifying a build so an older daemon cannot silently run it locally.
#### Requiring a verified tool
A compilation or job that needs a tool the fleet does not uniformly have can demand a worker
that has **verified** it:
```bash
rch exec --job --require-tool clang -- ./native_fuzz.sh
rch exec --require-tool cargo-xwin --require-tool llvm-ar -- \
cargo xwin build --locked --release --target x86_64-pc-windows-msvc --bin my-cli
```
The gate runs before any slot is reserved, and it is evidence-based: the worker
must have run the operator-declared probe (see
[Worker Config Example](#worker-config-example)) successfully. A project can
make the requirement a default for all of its jobs:
```toml
[jobs]
required_tools = ["clang"]
```
Project defaults ADD to `--require-tool` in job mode. These `[jobs]` defaults
apply only to jobs; explicit CLI requirements also apply to compilations while
preserving their classifier, Rust target and toolchain checks. Any required tool
makes execution strict remote, including when configuration forces local or
disables RCH. Both client and daemon must support the verified-tools endpoint;
an older daemon refuses the request instead of silently ignoring requirements.
A successful named probe establishes cached tool presence. It does not bind the
executable bytes, compiler version or SDK inventory used by a particular build.
**Exact Node and Bun versions.** `--require-tool` (alias `--require-toolchain`)
also accepts `node@VERSION` and `bun@VERSION`, matched against the version each
worker reports in its capabilities (`rch workers capabilities --refresh`):
```bash
rch exec --require-toolchain bun@1.3.2 -- bun test
rch exec --job --require-toolchain node@22.11.0 -- npm test
```
```toml
[jobs]
required_tools = ["node@22.11.0"] # job mode only, like named tools
```
The version must be a complete `MAJOR.MINOR.PATCH`; an optional prerelease or
build suffix must match exactly, and the `v` that `node --version` prints is
ignored. Ranges, partial versions (`22`, `22.11`) and aliases (`lts`) are
refused, and a worker that reports no version, or another one, is not
admitted. Like named tools, the check happens before reservation, makes the
run strict remote, and only filters: it never reorders workers. It compares
the version the worker's capability probe found. It does not install runtimes,
and it does not override a version manager (nvm, fnm, a login profile) that
picks a different binary when the job runs.
There is no `rust@VERSION`: a worker's default `rustc` says nothing about the
compiler a job selects. Pin Rust exactly with the toolchain instead, for
example `channel = "1.90.0"` in `rust-toolchain.toml` (or `RUSTUP_TOOLCHAIN`,
or `cargo +1.90.0`). RCH sends that toolchain to the worker, only selects a
worker on which `rustup run <toolchain> rustc --version` and
`cargo --version` succeed, and runs the build under that toolchain.
`rch admit [--job] [--require-tool NAME]... -- <command>` preflights the same
decision read-only — it syncs nothing and reserves nothing. Without `--job` the
preflight consults the compilation classifier, so a fuzz script answers
`local`; with it the classifier is bypassed exactly as `exec --job` does, and
the answer is `offload`. The classification facts are still reported, so a
compilation accidentally passed to `--job` still shows its family.
This static preflight reports requirements and classification; it does not
verify current worker readiness or establish native release qualification.
A required name that no worker has verified admits **no worker at all**,
including when the name is a typo. That is deliberate: silently dropping the
requirement would route the job to a machine that cannot run it, and job mode
returns the remote exit status verbatim — so the resulting failure would be
indistinguishable from the job's own. Selection reports
`capability_missing:tool:<name>:probe_failed` when a worker declared the tool
but its probe failed, and `...:not_declared` when nothing declared it.
### Same-identity job recovery
```bash
rch jobs # list durable local job leases
rch jobs attach <wrapper-id> --timeout-secs 300 # observe the original job
rch jobs cancel <wrapper-id> # cancel the identity-matched job
rch jobs recover <wrapper-id> --timeout-secs 300
```
For an unfinished detached retrieval, the current `[transfer]` configuration
can extend the original recipe's `max_transfer_time_ms`. Recovery persists a
larger positive budget under its exclusive journal lock; smaller values never
shorten the retained budget. Recovery extensions are bounded to 24 hours.
It keeps the original worker, build, output policy,
verification settings and ownership evidence, and never replays compilation.
Set `--timeout-secs` large enough for the entire recovery operation as well;
that observation deadline is separate from the transfer budget.
```toml
[transfer]
max_transfer_time_ms = 900000
retrieval_staging_dir = "/mnt/controller-storage/rch-retrieval"
retrieval_min_free_bytes = 67108864
```
`retrieval_staging_dir` is an absolute durable controller staging base, separate
from the worker's `remote_base`, the checkout and `TMPDIR`. Unset uses the
job-lease directory's `retrieval/` subtree. Before downloading, Unix retrieval
checks the selected uncompressed file size plus `retrieval_min_free_bytes`
against space available to the controller user. Windows-worker tar retrieval
also stages its bounded archive there and budgets both the archive and extracted
outputs while they coexist. This preflight cannot reserve
space against concurrent writers; a later storage failure remains retryable.
Failure retains the outputs and ownership and does not acknowledge completion.
Internal `.rch-clean-overlay-base.tar` source archives are excluded even when
recovering an older broad Make artifact recipe.
Recovery can redirect an old recipe's staging before any phase has published
or recorded a pending publication. It leaves the old stage intact and collects
the same worker outputs into the new location. Once publication begins, a
location change is refused to preserve its pending and published evidence;
extend its timeout or provide space on its existing filesystem instead.
Terminal acknowledgment still requires verified delivery and source release.
`rch --json jobs` reports valid leases alongside `journal_errors`. An unreadable
or malformed journal, or an entry observed as nonregular, makes `complete` false
and the command exits 1, while valid jobs remain visible in the same JSON document.
Each valid row
also reports `owner_presence` as `live`, `absent` or `unknown`; a false legacy
`wrapper_alive` value alone does not establish that an owner is absent. The listing
does not remove, repair or treat the unreadable entry as an absent owner. An
initially missing journal directory is an empty, complete listing.
The directory is opened once and journal names are read relative to its file
descriptor. Journal opens reject symlinks and do not wait for a FIFO writer;
the opened descriptor must represent a regular file before any bytes are read.
The directory root itself must be a directory rather than a symlink.
Initial attach/recover loads and recovery-owner reloads use the same regular-file,
nofollow, nonblocking reader. A changed parent directory is refused; a FIFO
cannot block these journal reads while waiting for a writer.
Detached recovery retains the opened directory and persistent lock through
journal publication and terminal reporting. Its clones share that ownership;
reads and atomic writes use the held directory. A replaced directory or lock
refuses acknowledgement. Failed recovery publications retain their exclusive
temporary bytes for diagnosis; recovery does not remove them. Ordinary running
wrappers keep their existing journal writer.
`complete` describes this scan, not an immutable ownership snapshot: entries
and regular-file contents can still change during observation.
If a journal path contains non-UTF-8 bytes, its error has a printable `path`
and an exact Unix `path_bytes` array. A job command whose output cannot be written
or flushed exits with an error rather than reporting successful delivery.
Completed/recovered reports include an `admission` reference from one committed
terminal observation: exact decimal build ID, wrapper, worker, journal path and
caller fingerprint, matching the ordinary execution reference. Unadmitted
terminal jobs have `admission: null`. Unknown exit, incomplete identity or
unacknowledged state cannot become a completed report. This reports journal
identity/custody; it does not replay the original execution envelope or attest
compiler, SDK, slots or artifact contents.
These commands never replay the original command. Recovery requires retained
identity and completion/retrieval evidence; ambiguous or missing evidence is
reported as an error. For a live wrapper stalled during artifact retrieval,
`recover_requested` means the request was recorded, not that outputs have arrived:
use `jobs attach` to observe completion. The wrapper cancels and reaps the old
transfer before retrying that retrieval phase. An absent wrapper can recover
outstanding outputs from its retained journal without starting another build.
Use a rebuilt daemon with the identity-aware job routes; older daemons are refused.
On POSIX workers, source ownership is saved before synchronization starts and
remains active through output retrieval, including after the SSH lock holder
disappears. Another job cannot overwrite an overlapping source root while that
ownership is unresolved; disjoint source trees can still run concurrently.
Uploads, verification, execution, and collection check the same ownership token.
Operations within one token are serialized. An uncertain preparation failure
ends that preparation; its token is drained and cancelled before a new attempt,
so an older upload cannot arrive after verification and change compiler inputs.
Recovery of an interrupted preparation drains surviving transfers and cancels
that token, so a delayed upload cannot mutate a tree after ownership is released.
Once execution may have started, recovery requires its exact completion record
or worker activity evidence that the original execution is gone; a missing
receipt alone does not prove completion.
If a worker loses its ownership registry or the local recovery recipe predates
this protocol, recovery refuses to infer ownership from the current source bytes.
Worker cache cleanup also respects retained source grants, including jobs whose
outputs are quiet but still await recovery.
`rch jobs recover` also reconciles an acknowledged terminal cancellation when
its retained recipe still owes worker settlement. A local `state=finished` and
`terminal_acknowledged=true` are insufficient by themselves: recovery verifies
the exact journal, failed result, resource retirement, worker records and drained
activity before recording `source_cancellation_settled=true`. This operation
does not reopen source trees or rerun the command. A retained `.cancelling`
marker may remain after success to fence delayed requests for the old token;
its presence alone does not block a new token. Missing or conflicting evidence
remains an error rather than permission to remove records.
Recovery records the source registry namespace as part of its durable recipe.
For older journals it first verifies exact worker evidence and checkpoints the
current or legacy binding before cancellation or cleanup. This keeps recovery
on the original registry and lock hierarchy instead of creating a contradictory
receipt in a newer namespace. Ambiguous or incomplete evidence remains refused;
cleanup also checks for competing lexical and physical owners.
New POSIX source-lock sessions send heartbeats every 10 seconds and expire after
90 seconds without a complete heartbeat, subject to worker process scheduling
and termination. Expiry retains durable ownership and leaves separately running
compiler/rsync activity to the normal recovery protocol. See the
[source namespace and terminal cancellation runbook](docs/runbooks/reliability-operations.md#source-namespace-and-terminal-cancellation-recovery-111)
for diagnostics, legacy registry handling, and the native-worker acceptance
still required for [#111](https://github.com/Dicklesworthstone/remote_compilation_helper/issues/111).
### Config + Diagnostics
```bash
rch config show|get|set|reset|init|validate|lint|doctor|edit|diff|export
rch doctor [--fix] [--dry-run]
rch doctor --reliability [--check-schemas] [--scope <scope>] [--strict|--lenient] [--json]
rch doctor --reliability --watch [--watch-interval N] [--transitions-only]
rch doctor --runbook RCH-Rnnn | --runbook-list
rch error explain <RCH-Ennn|RCH-Innn|RCH-Rnnn> | list [--category <name>]
rch self-test [--worker <id>|--all]
rch self-test status
rch self-test history --limit 10
```
### Fleet + Release + UX
```bash
rch update [--check|--rollback|--fleet]
rch fleet deploy|rollback|status|verify|drain|history
rch speedscore <worker>|--all [--history]
rch dashboard # alias: rch tui
rch web
rch schema export|list
rch completions generate|install|uninstall|status
```
### Agent Discovery Surface
For AI agents and automation, start with:
```bash
rch --robot-triage --json # quick_ref + recommended commands + health probes
rch capabilities --json # commands, aliases, env vars, exit codes, output formats
rch robot-docs guide # in-tool operating guide, no README lookup needed
rch --help-json workers/list # machine-readable help for nested command paths
```
`--json` uses the standard API envelope for command output; `--format toon`
emits the same data as TOON. The legacy `rch --capabilities` flag remains a
raw JSON shortcut for lightweight discovery.
---
## Configuration
Primary files:
- User config: `~/.config/rch/config.toml`
- Worker list: `~/.config/rch/workers.toml`
- Project override: `.rch/config.toml`
- Optional project excludes: `.rchignore`
Precedence (highest first):
1. CLI flags
2. Environment variables
3. Profile defaults
4. `.env` / `.rch.env`
5. Project config
6. User config
7. Built-in defaults
### Canonical Project Root
By default rch expects projects to live under `/data/projects` (with the `/dp`
alias symlink). Both roots are configurable, so repos under `~/code`,
`/workspace`, etc. work without relocating anything:
```toml
[path_topology]
canonical_root = "/home/me/code" # default: /data/projects
alias_root = "/home/me/code" # default: /dp (set equal to canonical_root if you have no alias)
```
Or via the CLI / environment (env wins over TOML):
```bash
rch config set path_topology.canonical_root /home/me/code
rch config set path_topology.alias_root /home/me/code
# or
export RCH_CANONICAL_PROJECT_ROOT=/home/me/code
export RCH_ALIAS_PROJECT_ROOT=/home/me/code
```
Empty strings are treated as unset (defaults apply). See
`docs/guides/configuration.md` (`[path_topology]`) for details.
### Loaded or High-Latency Workers
Increase bounded control-operation deadlines together when healthy workers need
more time for SSH setup, filesystem scans or lock acquisition:
```toml
[timeouts]
scale = 2.5 # default 1.0; finite values from 1.0 through 10.0
```
`RCH_TIMING_SCALE=2.5 rch exec -- cargo check` overrides the client policy for
one invocation. The daemon reads its own user configuration and environment at
startup; restart `rchd` after changing its policy. A project override or an
environment variable on `rch exec` does not reconfigure an already running
daemon. Inspect the effective client value with
`rch config get timeouts.scale` or `rch config show --sources`.
The scale extends slow-path control deadlines and bounded retry windows. The
shared timing policy caps each scaled control budget at one hour, including at
scale 1.0. Build and test execution time limits, signal-escalation grace,
polling cadence, disk thresholds, ownership checks and the proof required to
release a worker remain separate. Invalid, non-finite or out-of-range values
are configuration errors. See the
[configuration guide](docs/guides/configuration.md#timeouts) for scope and
precedence.
### Minimal Example
```toml
[general]
enabled = true
force_local = false
force_remote = false
socket_path = "~/.cache/rch/rch.sock"
log_level = "info"
[compilation]
confidence_threshold = 0.85
min_local_time_ms = 2000
remote_speedup_threshold = 1.2
build_slots = 4
test_slots = 8
check_slots = 2
# Optional additional build-disk headroom for each remote job (GiB).
# Include expected output growth and a safety margin. Zero disables it.
# disk_headroom_gib = 80
build_timeout_sec = 300
test_timeout_sec = 1800
bun_timeout_sec = 600
external_timeout_enabled = true
[transfer]
compression_level = 3
# Staging base for the transfer paths that do NOT mirror the client's layout:
# `--clean-overlay` roots, Windows workers, and `rch cache warm`. An ordinary
# `rch exec` mirrors the project under the worker's `[path_topology]
# canonical_root` instead, with the pooled Cargo target store inside that
# mirror — see `[remediation.pooled_target] store_base` below to place those
# stores on another filesystem.
remote_base = "/data/tmp/rch"
# Optional durable controller staging base (absolute, independent of TMPDIR).
# retrieval_staging_dir = "/mnt/controller-storage/rch-retrieval"
# Free-space reserve in addition to the selected uncompressed outputs.
# retrieval_min_free_bytes = 67108864
# Optional per-attempt source-sync cap. When unset, the default is payload-aware:
# 30 seconds plus one second per MiB, capped at one hour.
# sync_timeout_ms = 120000
# Abort a source sync after this many seconds of NO rsync output (dead channel /
# wedged rsync); a progressing transfer is never affected. 0 disables.
# source_sync_silence_timeout_secs = 120
# Explicit rsync binary. Unset lets RCH prefer a modern (3.x) rsync over the
# stock macOS openrsync automatically; RCH_RSYNC_BIN overrides this.
# rsync_bin = "/opt/homebrew/bin/rsync"
# Opt in to one supervised SSH connection throughout a Unix dispatch, shared
# by preflight, source locks, rsync, verification, execution, and retrieval.
# The foreground master is reaped when the dispatch and its cleanup finish;
# it does not leave a background ControlPersist process between invocations.
# Unset or zero keeps the existing one-shot exec paths. Windows workers and
# automatic recovery do not use this dispatch connection. Enable it for
# high-latency workers: without it every SSH step pays a fresh handshake.
# ssh_control_persist_secs = 60
adaptive_compression = true
verify_artifacts = false
max_transfer_mb = 2048
[remediation.pooled_target]
# Where pooled Cargo target stores are PLACED on the worker. Unset (the
# default) keeps them inside the project mirror, i.e. under the canonical
# root, on whatever filesystem holds it. Set an absolute path to move every
# pooled store to <store_base>/<project_id>/.rch-target-<worker>-pool-<key>,
# which is how a worker with a small root disk keeps multi-GB warm pools on a
# larger volume. `rch gc`, `rch cache status`, and the daemon sweep scan this
# root in addition to `remote_base` (with `reaper_max_cache_gb` applying per
# scan root). Ignored for Windows workers.
# store_base = "/bigdisk/rch-pools"
#
# Where the reaper SCANS for stale target dirs (not a placement setting).
# remote_base = "/data/projects"
#
# EXTRA roots `rch gc` scans, on top of the ones it derives itself
# (`remote_base`, `store_base`, and the worker temp base resolved exactly as
# the code that creates the dirs resolves it: $TMPDIR -> /data/tmp -> /tmp).
# Use this for a runtime root rch cannot know about, e.g. an operator build
# root on a mounted volume. Each entry must be absolute, `..`-free and free of
# shell metacharacters -- it is embedded in a remote command. `rch gc --root`
# adds one for a single run.
# gc_extra_roots = ["/mnt/big/rch"]
#
# Idle days before `rch gc` may collect a durable per-worker Cargo cache
# (`rch-cargo-cache-*`). 0 disables. Collection ALSO requires zero open file
# descriptors under the dir and no live process rooted at it; a gate that
# cannot be evaluated counts as "in use".
# gc_cargo_cache_idle_days = 14
[selection]
strategy = "balanced"
# Reserve this much free disk, then budget space for each concurrent slot (GiB).
min_free_gb = 10.0
disk_gb_per_slot = 10.0
# Critical when both this ratio and the absolute cap below are reached.
# disk_critical_free_ratio = 0.05
# Default cap in GiB; use "uncapped" for an unconditional critical ratio.
# disk_critical_ratio_cap_gb = 25.0
[self_healing]
hook_starts_daemon = true
daemon_installs_hooks = true
[alerts]
enabled = true
suppress_duplicates_secs = 300
```
When the Unix launcher enforces a compilation deadline, RCH preserves exit 137
and reports that the configured deadline expired. It does not retry on a larger
worker or fall back locally. Machine-mode `rch exec` reports
`outcome: "deadline_exceeded"`. This classification requires evidence from the
launcher; exit 137 or elapsed time alone does not establish a timeout.
For projects with large build outputs, set `compilation.disk_headroom_gib` in
`.rch/config.toml`. A value of `80` requires 80 GiB of additional space on the
worker's reported build filesystem. Selection requires a successful disk probe
within 90 seconds and subtracts budgets already held by active builds in this
daemon. Budgets survive daemon restart and remain held until the owning build
completes. Releasing a budget does not prove its output files freed any space:
the next budgeted admission requires a disk probe started after that completion.
A probe already in flight cannot reuse the earlier free-space reading. Retries,
queue recovery, and `rch diagnose` use the same requirement.
Smaller CPU-slot estimates and cache affinity cannot bypass it. Older daemons
reject the distinct budgeted selection endpoint rather than ignoring the
requirement.
This is admission accounting for the worker's reported canonical/alias build
roots, not a filesystem quota or a measurement of arbitrary custom target
mounts. Other dispatchers, undeclared jobs, and external writes are outside its
accounting. The full budget stays reserved even after a disk sample reflects
some of the build's output, so admission deliberately errs toward leaving extra
space. The default is `0`, retaining ordinary disk-pressure admission.
Without a declaration, the daemon still learns each project's footprint. For
every remote build that had a worker to itself (no other build from this
daemon overlapped there), it records how far the worker's free build-disk
space fell between the admission probe and the lowest probe seen while the
build ran. Footprints are kept per project and command class (`cargo test` is
learned separately from `cargo check`), for 30 days, in
`history.footprints.json` beside the build history. When some candidate
worker has room for the largest recent footprint plus 10% (at least 5 GiB),
after declared budgets and the remaining growth of builds already running
there, selection only considers those workers. If none has room, selection is
unchanged. Learned footprints are evidence, not a budget, so they never refuse
a build. Other dispatchers' builds can inflate a measurement and cache cleanup
can shrink one; declare `disk_headroom_gib` when you need a hard requirement.
Unix artifact downloads estimate the files matched by the retrieval filters
before transferring them. Their default total retry budget grows with that size
(30 seconds plus one second per MiB, adjusted for a slower `bwlimit_kbps`, up to
one hour). An explicit `max_transfer_time_ms` keeps its existing hard ceiling
and skips estimation. Rsync also stops after 30 seconds without network I/O.
If estimation fails, RCH logs the failure and retains the configured retry
budget. A literal `cargo build --bin NAME` or `--example NAME` retrieves only
the named payloads, not the pooled target's other cached outputs. Anything the
narrowing cannot parse exactly (unknown flags, shell expansion, other target
kinds, response files, custom target specs) keeps the broad per-profile filters,
which can include other cached build outputs in the same profile.
Windows workers return artifacts through tar in both quiet and interactive
execution. The transfer honors the requested output patterns and cache exclusions,
including custom Cargo target directories and explicit compiler outputs outside
`target/`. RCH inventories regular files, downloads only selected paths, and
validates the complete archive before writing outputs locally. Remote tar errors,
missing selected files, and unexpected archive members fail artifact retrieval;
successful downloads include file counts and a manifest for output validation.
Inventory and transfer use bounded deadlines, and cancellation reaps the transport
before recovery can begin. This transport requires GNU `find` from the worker's
Git for Windows installation and native `tar`.
Artifact staging retains the original caller directory as the source-protection
reference. Recovery restores that reference for each output phase, so an empty
staging directory cannot make source files eligible for a broad output pattern.
Built-in worker selection defaults to `balanced`, which blends speed, load,
health, and cache affinity. Use `priority` only when you want explicit
worker-priority control, and `fair_fastest` when you want extra load spreading.
Worker capacity is capped at the configured `total_slots` and reduced to
`floor((free_disk_gb - min_free_gb) / disk_gb_per_slot)` as disk fills, with a
minimum of zero. Unknown disk telemetry keeps the configured ceiling. Existing
builds retain their reservations when capacity drops; new work waits for space.
`rch workers list` reports the effective total when the daemon is reachable.
Change the budget with `rch config set selection.disk_gb_per_slot 10` and restart
the daemon to apply it.
Separately, on build filesystems of at least 50 GiB, rchd marks a worker Critical
when free space falls to 10 GiB, or when **both** the free ratio is at or below
`selection.disk_critical_free_ratio` (default `0.05`) and free space is at or below
`selection.disk_critical_ratio_cap_gb` (default `25.0` GiB). These defaults also
apply when the dispatcher has no `config.toml`. A 900 GiB disk with 30 GiB free
retains disk-budgeted capacity despite its low free percentage; the same disk
with 8 GiB free stays Critical. Existing small-filesystem and `/tmp` rules still
apply. A noncritical classification alone does not admit a
build: fresh storage evidence, active reservations, the authorized build budget
and safety buffers must still fit.
Explicit caps such as `15.0` or `30.0` and custom critical ratios remain honored.
Set `disk_critical_ratio_cap_gb = "uncapped"` to enforce the ratio independently
of absolute free space. A finite cap must be at least 10 GiB; invalid numeric
caps fall back to the stricter uncapped rule with a daemon warning. Inspect or
change the settings with `rch config get selection.disk_critical_ratio_cap_gb --sources`
and `rch config set selection.disk_critical_ratio_cap_gb 25`. The `get` and `set`
commands also accept `selection.disk_critical_free_ratio`.
Both keys take effect when rchd restarts, and its startup log reports the
effective disk policy. The cap does not change per-slot allowances, protected
free-space floors, or authorized build-storage budgets.
Balanced selection also favors free disk before a worker reaches the admission
floor. `selection.weights.disk` (default `0.2`, range `0` to `1`) adds disk
headroom credit after the other score adjustments. Equal ample headroom preserves
the existing worker order. Credit rises from zero at `min_free_gb` to full credit
at 25% free disk, with at least one slot's disk budget above the floor on small
disks. Unknown measurements receive full credit; the admission gate still handles
missing telemetry. Affinity pins and explicit worker requests retain their
existing behavior. Set the weight to zero to disable disk ranking independently
of disk admission and slot limits, then restart the daemon to apply the change.
`rch workers list` shows daemon-reported free disk and pressure. `rch workers probe`
shows fresh disk measurements alongside latency and labels any cached daemon
pressure separately. JSON and TOON include numeric free GiB and free ratios;
unavailable measurements remain `null` (displayed as `unknown` in text).
### Worker Config Example
```toml
[[workers]]
id = "css"
host = "203.0.113.20"
user = "ubuntu"
identity_file = "~/.ssh/id_rsa"
total_slots = 32
priority = 100
tags = ["fast", "ssd"]
# Verified named tools. Each entry is a FIXED argv the worker runs as the
# configured user; a zero exit marks the tool present, anything else marks it
# absent. Unlike `tags`, which are an unverified naming convention, these are
# evidence — which is what lets `--require-tool` gate worker selection.
tools = [
{ name = "clang", command = ["clang", "--version"] },
{ name = "ld.lld", command = ["/usr/bin/ld.lld", "--version"] },
]
```
Names must be ASCII letters, digits, `-`, `_`, `.` or `+`: a name carrying a
space or `=` could not survive the probe's fact format and would be read back
as a different name, so it is refused when the config loads rather than
becoming a capability lie. `rch workers capabilities --refresh` reports the
results under **Named tools** as `verified:` and `failed:` — a declared probe
that fails is a different operational state from one that was never declared,
and only the first points at a broken worker.
There is deliberately no general remote-shell probe API. A caller that could
ask a worker to run an arbitrary command "to check for a tool" would be a
remote execution primitive wearing a capability-probe hat; declaring the argv
in operator config keeps the set of probe commands finite and reviewable.
### Worker Execution Policy (Linux workers)
A worker can lower the priority of the builds RCH runs on it, or confine them
to a systemd slice, independently of `rchd`'s own service settings. The policy
is set on the worker, in exactly one of these files:
- `/etc/rch/worker-execution.conf`, owned by root, or
- `~/.config/rch/worker-execution.conf` of the SSH user, owned by that user or
by root.
```ini
version=1
nice=10
systemd_slice=jobs.slice
systemd_manager=user
```
`version=1` is required, plus at least one of `nice` (-20 to 19) and
`systemd_slice` (an existing, active `*.slice` unit). `systemd_manager` (`user`,
the default, or `system`) names the manager that owns the slice and is only
valid together with `systemd_slice`. Every setting in the file is enforced
before the build starts. The launch is refused, never run unconfined, when the
file is unsafe (not a regular single-link ASCII file of at most 8 KiB, writable
by group or others, wrongly owned), has unknown or duplicate keys, or exists in
both places, and when enforcement fails (no `busctl`, `systemctl` or `timeout`,
an inactive slice, a failed priority change, a cgroup placement it cannot
confirm, or a non-Linux worker). The launcher writes the cause to its stderr as
`RCH_WORKER_POLICY_REFUSED:<reason>`, and what it applied as
`RCH_WORKER_EXECUTION_POLICY:nice=…,slice=…,manager=…,scope=…`.
With a slice, RCH attaches the build's existing process-group leader to a
transient scope in that slice, so the watchdog, cancellation and recovery keep
working, and cleanup covers the whole cgroup. Without a policy file, builds run
as before.
The policy covers the tracked build payload only. Source sync, pre-launch
package bootstrap and untracked execution are not confined, so it does not on
its own make a shared developer machine safe to admit as a worker (#104).
---
## Output Modes
RCH auto-selects output mode by context:
- `hook`: strict JSON for hook protocol
- `machine`: explicit machine output (`--json`, `--format`)
- `interactive`: rich terminal rendering
- `colored`: ANSI-only when forced without TTY
- `plain`: text fallback
Environment controls:
- `RCH_JSON=1`, `RCH_HOOK_MODE=1`
- `NO_COLOR=1`, `FORCE_COLOR=1`, `FORCE_COLOR=0`
- `RCH_OUTPUT_FORMAT=json|toon`, `TOON_DEFAULT_FORMAT`
JSON responses use a stable envelope (`api_version`, `timestamp`, `success`, `data`, `error`).
---
## Placement Controls
Worker placement, strict-remote, queue, wait-timeout, visibility, and target-dir
behavior are first-class, canonical controls — not folklore. The authoritative
list is discoverable at runtime (`rch capabilities --json`), and the resolved
plan for any command is shown by `rch diagnose <command> --json` under
`data.placement` (and in the human `Placement Controls` section):
| Control | Env (aliases) | Effect |
|---|---|---|
| Requested worker | `RCH_WORKER` (`RCH_WORKERS`) | Request specific worker(s) by id. Still passes capability/admission checks; an inadmissible requested worker is **refused** with a stable `RCH-Innn` reason code and a next action — never silently swapped. |
| Requested profile | `RCH_PRESET` | Named execution profile (recorded as `requested_profile`). |
| Strict remote (fail-closed) | `RCH_REQUIRE_REMOTE`; per call `rch exec --require-remote` | Refuse local fallback (proof mode). Takes precedence over `RCH_FORCE_REMOTE`. |
| Force remote (fail-open) | `RCH_FORCE_REMOTE` | Always attempt offload (bypass local-time/speedup gating) but still fail open to local. Distinct from `RCH_REQUIRE_REMOTE`. |
| Queue when busy | `RCH_QUEUE_WHEN_BUSY` (default `1`) | Wait for a busy worker instead of falling back to local. Set `0` to disable. |
| Wait timeout | `RCH_DAEMON_WAIT_RESPONSE_TIMEOUT_SECS` (`RCH_DAEMON_RESPONSE_TIMEOUT_SECS`) | Max seconds to wait for a queued worker: default 330, or 6 h for `rch exec --job`. The daemon caps a compilation's wait at `[queue] timeout_secs` (300) and a job's at 24 h. |
| Visibility | `RCH_VISIBILITY=none\|summary\|verbose` (`RCH_QUIET`, `RCH_VERBOSE`) | Hook output verbosity. |
| Target dir | `RCH_DISABLE_TARGET_REUSE` | Legacy unique-per-job remote target dir instead of the pooled, reuse-friendly dir. |
| Source-sync timeout | `RCH_SYNC_TIMEOUT_MS` | Per-attempt source-upload timeout in milliseconds (1000..=3600000). Unset uses the payload-aware default. This does not change remote Cargo or artifact-return timeouts. |
The resolved plan reports `requested_worker`, `requested_profile`,
`effective_worker`, `strict_remote_policy`, `queue_policy`, `visibility_mode`,
`wait_timeout_ms`, `target_dir_policy`, the requested-worker admissibility
outcome, and a `diagnostics` list. Any control value that cannot be applied as
written (an unrecognized value or a superseded alias) surfaces a diagnostic
rather than being silently ignored.
The simulated selection also lists every worker the daemon considered, under
`data.worker_selection.inventory` (human output: `Inventory:` under `Worker
Selection (simulated)`). Each entry is `selected`, `eligible` (passed every
gate, ranked lower), `fallback_candidate` or `excluded`, with the selector's
reason and reason codes, for example `available slots 2 < 4`
(`slots.insufficient`), `not in the requested worker set`
(`selection.not_requested`), or a project exclusion such as
`project.live_checkout_fenced`. A daemon older than the client reports no
inventory after a successful selection.
---
## Monitoring and Observability
RCH exposes observability through daemon APIs and metrics:
- daemon health/readiness endpoints
- Prometheus metrics collection
- opt-in OTLP export of daemon request-duration metrics
- telemetry-backed worker SpeedScore history
- queue/build history, active alerts, cancellation metadata in status APIs
- an opt-in **tailnet status API** (`[api] bind = "tailscale"`): the daemon's
full `/status` JSON over TCP, bearer-token gated, bound only to loopback or
Tailscale addresses — so agents on other machines can ask a dispatcher "why
are you local-only right now?" without ssh
- the **fleet dashboard** under `dashboard/` (encrypted static console + an
agent endpoint `GET /api/fleet?view=problems|diagnose|help`); `rch web` opens it
Enable OTLP metrics in the daemon's environment with `RCH_OTEL_ENABLED=1`
and `RCH_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317` (OTLP/gRPC).
The endpoint falls back to `OTEL_EXPORTER_OTLP_ENDPOINT`. `OTEL_SERVICE_NAME`
defaults to `rchd`; `RCH_OTEL_EXPORT_INTERVAL_SECS` defaults to 30.
`rch_request_duration_seconds{entrypoint="rchd_api"}` records finite Unix-socket
request handling after parsing, including errors and cancellation. Event streams
and the separate HTTP API are excluded. The same observations reach Prometheus,
including when OTLP is disabled. Shutdown drains request tasks before a
best-effort final export. The banner reports exporter configuration, not collector
connectivity.
The same opt-in settings enable direct OTLP verdict export for single-shot
`rch doctor --reliability`, including under `--quiet`. Its service name defaults
to `rch`. Each report emits one `rch_doctor_verdict_total` observation, labeled
with the actual verdict and scope (combined scopes use `other`), and flushes
before returning its original exit code. Reports whose `daemon_unreachable`
flag is true also increment `rch_doctor_daemon_unreachable_total` once, even
when several diagnostics report the missing daemon. The asynchronous
`daemon_status` and `repo_convergence` probes also emit
`rch_doctor_probe_duration_seconds{probe,result}` once on completion, RPC error,
or timeout. Durations measure each probe's execution, not time spent waiting
to join other probes. Skipped probes emit nothing; panic/cancellation and
helper/ownership probe durations are not yet exported. With `--fix`, each
remediation step also emits `rch_doctor_fix_steps_total` and
`rch_doctor_fix_duration_seconds`, labeled by its actual outcome: `applied`,
`already_satisfied`, `would_apply`, `manual`, or `failed`. Dry-run previews
remain `would_apply`; they are never counted as applied changes. Durations
measure step execution, excluding the initial shared configuration load.
These CLI metrics are not forwarded to the daemon's Prometheus endpoint.
Hook/watch telemetry, remaining doctor event metrics, and trace/log export
remain unfinished.
Quick checks:
```bash
rch status --workers --jobs
rch speedscore --all
rch doctor --json
```
On Linux boxes configured with `[general] role = "dispatcher"`, status and doctor
also report unmanaged local Cargo/rustc processes. The warning remains visible
while those processes run, including when the daemon is unavailable. Check
`rch shim status`, PATH order, and absolute-path toolchain Cargo invocations.
Processes descended from `rch` or carrying `rch`'s internal local-fallback marker (`RCH_CARGO_WRAPPER_BYPASS=1` with `RCH_CARGO_WRAPPER_ORIGIN=rch`) are excluded; a bare caller-set `RCH_CARGO_WRAPPER_BYPASS=1` is not excluded.
An alarm event is logged once per observed episode and clears after a successful
scan finds no unmanaged compiler processes. Status and doctor share a locked
one-byte latch at `$XDG_CACHE_HOME/rch/local-build-alarm` (normally
`~/.cache/rch/local-build-alarm`). Scan failures preserve the previous episode;
latch failures are reported without hiding current builds. Reliability doctor
includes this check in the `triage` scope and its default `all` scope.
---
## Testing and Validation
Workspace checks:
```bash
cargo fmt --check
cargo check --workspace --all-targets
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
```
Reliability and E2E suites are provided under:
- `tests/`
- `tests/e2e/`
- `rch-common/tests/` (contract/reliability/perf suites)
If you are running CPU-intensive validation manually and want explicit offload:
```bash
rch exec -- cargo check --workspace --all-targets
rch exec -- cargo test --workspace
rch exec -- cargo clippy --workspace --all-targets -- -D warnings
```
When local fallback is not acceptable, set `RCH_REQUIRE_REMOTE=1` on the
`rch exec` process and keep the build command as direct argv:
```bash
RCH_REQUIRE_REMOTE=1 rch exec -- cargo test --workspace
RCH_REQUIRE_REMOTE=1 rch exec -- cargo clippy --workspace --all-targets -- -D warnings
```
`rch exec --require-remote -- <cmd>` asks the same for one call without
touching the environment. Because it is always the caller's own request (the
cargo shim exports `RCH_REQUIRE_REMOTE=1` into every agent), it also outranks
config that would run the command locally (`force_local`, `enabled = false`,
the execution allowlist), as `--worker` does. A non-compilation command is
refused with `RCH-E301` instead of running locally; to run one on a worker, use
`rch exec --job -- <cmd>`.
### Parallel dependency uploads
Ordinary Cargo builds on Unix workers can upload up to eight independent
dependency roots concurrently while holding their durable source grant. Every
upload keeps checksum comparison and its existing transfer timeout, progress
reporting, and receiver safety checks. Source verification, compilation, and
retirement wait for all admitted uploads to finish. A failed upload stops new
admission, drains every upload already admitted, and enters the existing durable
preparation-cancellation protocol before another build can reuse those sources.
Overlapping source roots keep their original order. A worker destination with a
symlinked ancestor or an ambiguous physical directory identity uses the
exclusive upload path. Clean-overlay archive materialization and Windows
transfers keep their existing ordering. A positive `bwlimit_kbps` also keeps
uploads sequential so concurrency cannot multiply the configured bandwidth cap.
### Exact source-content receipts
Proof-oriented callers can require a single-worker, fail-closed source transfer
and retain the exact regular-file bytes admitted on that worker:
```bash
RCH_REQUIRE_REMOTE=1 rch exec --source-content-receipt -- \
cargo test --locked --workspace
```
Receipt mode resolves the active Cargo path-dependency closure, stages its
roots beneath the configured worker build base, transfers with checksum
comparison, and reopens every selected file before and after the command.
Preflight initializes the isolated staging topology and skips canonical worker
repository convergence. The emitted
`rch.source_content_receipt.v1` JSON binds the worker and build IDs, exact
command digest and exit code, transfer filter policy, per-root file paths,
lengths, executable bits, SHA-256 digests, and content roots. Any unproved
dependency closure, transfer delta, worker verification error, local lasting
change, retry, or local fallback refuses the invocation instead of emitting a
receipt. Receipt mode requires the Unix rsync transport.
While holding the source grant, receipt verification uses rsync to check the
selected namespace and metadata, then independently verifies every worker file
against the captured SHA-256 manifest and recaptures the local inputs. This
avoids a duplicate full-file rsync checksum pass during verification. Uploads
still use checksum comparison, and verification without a source grant retains
the checksum barrier. Equal size and timestamps never establish byte identity.
Clean-overlay builds can request the same byte-level proof:
```bash
rch exec --base HEAD --clean-overlay --overlay-path src/lib.rs \
--source-content-receipt -- cargo test --locked
```
The manifest covers the complete selected Git archive plus the exact files
transferred by the explicit overlay filters. Unselected working-tree edits
never become receipt inputs. `--no-overlay` proves the committed tree and its
pinned submodules; each `--dependency-base PATH=REV` adds its own selected root.
The filter policy records the commit, tree, invocation directory, pinned
submodules, overlay paths, and fingerprint separately from the rsync rules,
because those rules select only the live overlay. Every selected
file is checked on the worker before and after execution, including committed
files that are absent or dirty in the local checkout. Receipt mode preserves
the clean-overlay source/target pair's stable paths and warm compiler cache.
An additional read-only namespace check rejects extra worker files, symlinks,
or directories, including in base-only runs. Only RCH-owned runtime paths and
the exact root archive transport file are exempt; selected source inputs that
overlap those exclusions refuse before upload. Missing selected paths also
refuse. Commands that leave source changes or outputs outside the owned runtime
paths therefore cannot produce a successful source-content receipt.
On Linux with target reuse enabled, supported Cargo commands use an owned
source/target pair. The source layout preserves sibling dependencies, selected
manifests, and enclosing workspace metadata under a stable
`source-content-pair-<key>/projects/` container. Every invocation materializes
fresh source files there while holding the pair and its complete source grant.
Uploads use checksums without deleting entries or running cache sweeps; the
read-only rsync barrier still rejects surplus selected files. Cargo's target
and build directories are bound to the granted pool outside that container.
An unchanged file can reuse its previous compiler-visible timestamp only when
the previous attempt completed its source-content verification. An interrupted
or unverified attempt forces fresh timestamps beyond the pool's artifacts on
the next invocation. After artifact delivery, the source container moves to
its recorded `.retained-<attempt>` sibling using Linux no-replace rename. The
job journal retains both paths before acquisition, so recovery can reconcile
an interrupted move without replacing an existing retained tree. Python 3 and
the Linux rename capability are checked before source ownership is acquired.
This owned route validates Cargo manifests and configuration against the
captured transfer inventory. Relative and absolute `--manifest-path` selectors
map into that inventory. File-based `--config`, custom target-specification
files, directory-changing wrappers, and caller overrides of managed output
paths or Cargo home currently refuse before execution. Other Unix platforms,
invocations with `RCH_DISABLE_TARGET_REUSE=1`, and external Cargo drivers such
as Nextest, Zigbuild, and xwin retain invocation-unique staging paths. The
owned route currently covers the built-in build, test, check, documentation,
benchmark, and verified packaging families, plus Clippy.
The receipt is emitted after the remote command and its post-command source
barrier, but before ordinary artifact retrieval completes. Artifact-grade
callers must therefore retain both the receipt and the later successful
retrieval/terminal transcript; the receipt alone is not proof that build
artifacts reached the caller. A recursive local mutation watcher is still
needed when transient edit-and-restore (ABA) detection is part of the caller's
source-stability contract.
### Clean Git overlays for shared working trees
`rch exec` can build an immutable committed tree plus an explicit, repeatable
set of local paths without transferring unrelated working-tree changes:
```bash
RCH_REQUIRE_REMOTE=1 rch exec \
--base HEAD \
--clean-overlay \
--overlay-path src/lib.rs \
--overlay-path tests/focused.rs \
-- cargo test --test focused
RCH_REQUIRE_REMOTE=1 rch exec \
--base HEAD \
--clean-overlay \
--no-overlay \
-- cargo check --workspace --all-targets
```
The client resolves `--base` to a commit object, streams that Git archive into
a fresh isolated worker path, and then uploads only the literal repository-
relative `--overlay-path` selections. Modified and untracked files are
supported, together with selected deletions of tracked regular files and
directories containing only regular files. A directory selection also includes
tracked files and subdirectories removed within that directory. Deleted entries
are omitted from the composed archive and source receipt; no deleting rsync is
used, and cached source bases remain intact. Their absence is rechecked before
execution, so a deleted input that reappears invalidates the admitted selection.
For a rename, select both the old and new paths, or select their containing
directory. Selecting only the destination keeps the unselected original from
the base. For example, after renaming `src/old.rs` to `src/new.rs`:
```bash
rch exec --base HEAD --clean-overlay \
--overlay-path src/old.rs --overlay-path src/new.rs \
-- cargo check
```
Missing paths that were not present in the base, absolute/traversing paths,
Git metadata, non-ASCII/backslash/control-character paths, case-only or otherwise
ambiguous filesystem spellings, overlay symlinks, and Git archive
`export-ignore`/`export-subst` attributes fail closed. A selected directory may
become empty through admitted deletions; newly selected empty directories remain
unsupported. Exactly
one of one-or-more `--overlay-path` options or `--no-overlay` is required.
Clean-overlay mode also implies remote-only execution even when
`RCH_REQUIRE_REMOTE` was omitted, so a worker or transfer failure never falls
back to the ambient local tree.
Explicit clean-overlay execution also admits the read-only `cargo fmt --check`
diagnostic, which the ordinary interception classifier intentionally leaves
local.
On Unix workers, clean-overlay commands can start below the Git root. RCH
materializes the entire selected repository and executes from the corresponding
selected subdirectory. Cargo discovers the nearest selected ancestor manifest
and toolchain; dirty or untracked nearer files do not change that choice.
Relative command arguments, `--manifest-path`, `--dependency-base`, target
destinations, and job result directories retain their caller-directory basis.
`--overlay-path` always remains relative to the Git root:
```bash
cd crates/app/src
rch exec --base HEAD --clean-overlay --overlay-path crates/app/src/lib.rs \
-- cargo test --manifest-path ../Cargo.toml
```
The invocation directory must exist in the selected base or overlays, with an
unambiguous spelling and no selected file or symlink ancestor. Trailing-space
directory names are refused. The receipt names the invocation directory, and
recovery keeps command/output locations separate from the full source roots.
Named Cargo JSON outputs use the full selected source root to map ancestor
manifests back to the caller, including after detached recovery. Source leases
and retirement retain the complete selected roots. Disk-growth accounting
still covers the full primary source root.
Initialized Git submodules are materialized recursively at the exact gitlink
commits in the selected base. Their dirty working files and current checkout
revisions do not affect the selected bytes. Each submodule must have the pinned
commit locally; RCH never fetches or updates a checkout during admission. Missing
pins, uninitialized or symlinked submodule roots, archive-transform attributes
or NUL-containing attribute files in any selected repository, and overlays
inside or containing a submodule fail closed. All clean-overlay compilers,
including direct `rustc`, validate selected symlinks before upload: targets must
exist inside the selected source inventory without escapes or cycles. The source
fingerprint and execution receipt bind every submodule path,
commit, and Git tree. Admission allows at most 1,024 submodules and 32 nesting
levels, including queued descendants. Composed archives preserve executable
permissions and symlinks and refuse case-aliased destination paths.
Admission reads pinned symlink targets, Cargo metadata, and archive attributes
in Git object batches per selected repository. Symlink resolution still uses
the complete selected inventory, so a linked Cargo configuration is read from
its pinned target even when that target has a different filename. Every batch
response must match the requested object ID, blob type, and exact byte framing;
missing objects, malformed responses, and bounded-resource failures refuse
admission. This reduces Git process startup work without consulting dirty
checkout contents or introducing a timestamp-based source cache.
On Unix workers, pooled Cargo builds use a stable source path paired with a
versioned target cache. Jobs sharing that pair wait until the preceding job
has finished execution, artifact retrieval, and source retirement. Different
pairs remain independent. Source timestamps are refreshed beyond cached
artifacts so changed archive or overlay contents cannot appear older than a
previous build; registry and Git dependency caches remain reusable.
These paired builds also retain immutable Git archives in a private directory
inside the owned target pool. Each selected repository has a separate slot, so
primary and sibling dependency archives can stay warm together. Before reuse,
the worker checks the bounded cache header against the caller's requested
archive. A different identity is a miss without reading the old payload. A
matching archive is copied into the fresh source tree while its length and
SHA-256 are verified in one pass against the archive constructed by the caller;
only a verified private copy can proceed to extraction. A verified hit skips
the base archive upload; local archive creation
and source-content verification still run. The cache admits at most 64 slots
and 1 GiB including pending publications, and preserves the configured worker
free-space reserve. Capacity limits or unavailable Python/layout support use
the ordinary upload path. Partial or corrupt cache files never count as hits.
Cache files are excluded from artifact retrieval, including priority includes
and recovery. Workers without durable paired source/target ownership continue
to use ordinary archive uploads.
Clean-overlay execution also pins Cargo's `build.build-dir` to its assigned
target directory, overriding ambient and command-line Cargo configuration.
Explicit `--build-dir` options are refused. This placement applies to isolated
builds too, so disabling target reuse cannot inherit shared intermediates.
Managed placement currently supports built-in Cargo build commands and Clippy.
Other compiling Cargo subcommands, including Nextest and Zigbuild, are refused
in clean-overlay mode.
An interrupted pair retains its ownership marker and refuses further reuse,
even if its SSH lock connection disappears. Inspect the prior job before
recovering that pair, or use `RCH_DISABLE_TARGET_REUSE=1` for a fresh isolated
build. Non-Cargo Windows clean-overlay jobs also use isolated targets. The ownership
marker protects against process interruption; it is not a power-loss durability
guarantee.
Clean-overlay materializes the primary Git repository and explicitly bound
sibling repositories. In-repository Cargo workspace members are present in the
archive. Before Cargo starts, selected
manifests, configuration, symlinks, and command-line path overrides are checked
against the selected Git base and overlays. Escaping paths are refused with
`RCH-E413`; retained sibling directories on the worker cannot supply those
dependencies. To include a sibling Git repository, bind its revision explicitly:
```bash
rch exec --base HEAD --clean-overlay --no-overlay \
--dependency-base ../dep=HEAD -- cargo test
```
Repeat `--dependency-base PATH=REV` for every external root, including transitive
dependencies. Each revision resolves to a commit before worker selection; dirty
files in those repositories are excluded. The selected repositories must be
siblings of the primary Git root. They are staged under one owned container,
preserving `../dep` references, and the execution receipt names each commit,
Git tree hash, and remote path. All roots share the same execution lease and
are retired together. Cross-repository symlinks and non-sibling layouts are
currently refused.
Manifest semantics follow the selected entrypoint's workspace members and local
dependency closure, including dev, build, target-specific, optional, inherited,
patched and replaced path dependencies. Unreferenced Cargo manifests remain
transferred fixture bytes; naming the same fixture as a member, dependency or
entrypoint activates the same containment and source checks. Workspace excludes
are literal directory prefixes; explicit members override them, while excluded
default members stay excluded. Exclusions never hide actual path dependencies.
Legacy Cargo dependency-table and package-table aliases retain these checks.
Workspace pointers in selected ancestor packages activate their captured root
too; absolute, absent, aliased or non-workspace targets refuse before Cargo.
Selected manifest-file symlinks are refused because resolving their bytes must
not change Cargo's relative source base; unreferenced aliases remain data.
All inventory paths/symlinks and selected Cargo configuration retain their
containment checks. File-based `--config`, configuration includes, and changes to Cargo's
working directory through `-C` or wrapper `--chdir` are refused. On the worker,
selected ancestor configuration within the primary Git root is allowed;
Cargo-home configuration and configuration above that root are refused before
Cargo starts, even if harmless. Cargo clean-overlay jobs require a Unix worker for this
check. These checks do not sandbox arbitrary file reads by build scripts.
The client re-fingerprints overlays after upload and refuses execution if their
contents changed during admission or transfer.
Do not batch several Cargo commands behind a shell wrapper such as
`rch exec -- bash -lc "cargo test ... && cargo test ..."`. Shell-wrapped
commands are classified as non-compilation for hook safety. Under
`RCH_REQUIRE_REMOTE=1`, RCH refuses that local fallback before executing the
shell and reports `RCH-E301`; without the env var, ordinary non-compilation
commands may still run locally. For several focused checks, run separate direct
`RCH_REQUIRE_REMOTE=1 rch exec -- cargo ...` invocations.
---
## Security Model
- Transport uses SSH.
- Worker commands are constrained to classified execution paths.
- Sensitive field masking and structured error taxonomy are built in.
- `rch update` enforces the release's SHA-256 checksums and verifies the
archive's `.minisig` against a public key pinned in the binary (minisign key
`69B3955C8D2E62A8`, which signs every release since 2.1.0; 2.0.x and earlier
used `1BBD79B28BF718D0`); a bad signature
aborts the update. A Sigstore bundle is also verified when the release ships
one and `cosign` is installed. `--skip-verify` bypasses signature checks only.
`install.sh` still verifies checksums only.
- Hook path remains fail-open to avoid deadlocks/stalls.
Operational recommendations:
1. Use workers you control.
2. Use dedicated SSH keys for worker access.
3. Keep workers patched and isolated.
4. Enable telemetry/alerting for production-like use.
---
## Limitations
- Designed around SSH-based Linux worker environments.
- Tooling assumptions are strongest for Rust and selected build/test commands.
- Remote performance gains depend on network + worker capacity + project shape.
- The fleet dashboard (`dashboard/`) needs Node and a static host (Vercel/Pages); its live path needs a Vercel Blob store and `rchd`'s `[api]` listener on a tailnet.
---
## FAQ
### Does RCH block my command if the daemon/workers fail?
No. It fails open and allows local execution.
### Can I force local or force remote per project?
Yes, via `.rch/config.toml` (`general.force_local` / `general.force_remote`).
### Is queue/cancel supported?
Yes. Use `rch queue` and `rch cancel`.
### Can I inspect why a command is or is not intercepted?
Yes. Use `rch diagnose "<command>"`.
---
## About Contributions
Please don't take this the wrong way, but I do not accept outside contributions for my projects. You can still open issues and PRs for discussion/proof-of-fix, but I review and re-implement changes independently.
---
## License
MIT License **with an OpenAI/Anthropic rider** — this is not ordinary
OSI MIT: no rights are granted to the restricted parties named in the
rider. See [LICENSE](LICENSE) for the exact, controlling terms
(SPDX: `LicenseRef-MIT-OpenAI-Anthropic-Rider`).