dev-cache

# Cache Control ## One authority `config/cache.toml` is the only cache inventory and policy. Every independently accounted cache has exactly this common contract: - `description`: what expensive work the bytes preserve; - `scope`: `disk`, `docker`, or `tart`; - `max_size_bytes`: the usage ceiling that triggers enforcement; - `warm_size_bytes`: the retained target after crossing the ceiling; - `prune_strategy`: `lru`, `generational`, `ephemeral`, `none`, `docker`, or `tart`. Age and count fields are optional refinements. They never replace the common contract. Do not introduce storage-availability rails, warning thresholds, or a second config table. Cache decisions are based exclusively on owned usage. ## Typed boundary Programs use `CacheRegistry`, `CacheRequest`, `CacheOperation`, and `CacheMutationResult` from `capsem_builder.cache`. They select a stable cache ID and do not branch on storage mechanism or discover backend locations. The registry owns backend dispatch: - `DiskBackend` inventories repository cache stages through `CachePaths`; - `RuntimeBackend` inventories and mutates Docker or Tart through their native adapters; - Docker-on-Colima is still the `docker` cache. Colima is an implementation detail of that backend, never a separately billed cache; - Docker image repositories are child cache IDs but use the same contract and request/result types as every other owner. A producer that must place disk bytes asks `CachePaths` or the gate's `cachelayout.stage_path()` for its configured stage. It must not reconstruct a path from repository literals, environment variables, or a backend name. A consumer that only inspects or controls a cache uses `CacheRegistry` and never receives its path. A shared cache stage must not contain or be contained by a prefix export. Exports are replaceable product views; shared stages are retained authorities. Overlapping them makes a successful export erase its own receipts and traps an exact-source run in a repair loop. VM generations live in the `assets` cache stage, while `cache/target/assets` remains only the assembled runtime view. Pinned external executables use `CachedToolPolicy` and `materialize()`. The policy declares one HTTPS URL and SHA-256 per host platform; the primitive downloads once, verifies before atomic publication, installs mode 0555, and returns a typed hit/miss result. A subsystem must not invent its own downloader or tool directory. Short-lived results from mutable services use `CleanVerdict`, `reusable()`, and `record_clean()`. The subject digest covers the complete typed policy and exact input bytes. Only success is recorded; malformed, future-dated, expired, or non-matching receipts are misses. The owning cache stage supplies the maximum age, count, warm size, and maximum size. `CachePaths` is loaded through `load_paths()`. The loader resolves the policy-owned `authority_environment`; a gate prefix exports that variable once so every producer sees the outer shared cache even when its output path crosses a prefix symlink. Producers never read or interpret the environment variable themselves. With no explicit authority, `load_paths()` resolves a linked Git worktree to the checkout that owns the common `.git` directory. All branches of one local repository therefore share one cache inventory. Do not create a cache beside each worktree or pass a worktree path as implicit storage. Administrative cleanup of a retired location uses the typed CLI's explicit `--repository` with `--policy-repository`; ordinary operators stay on `just cache`. All models are strict, frozen Pydantic models with `extra="forbid"`. Keep every module at or below 300 lines and put tests in the matching test module. ## Operator interface The public entry point is `just cache`. These are the complete supported inspection and lifecycle commands: ```bash just cache stats # usage, warm/max contract, state, description just cache stats --json # typed machine-readable inventory just cache stats --offline # disk inventory without native runtime calls just cache contract <cache-id> # the five-field common contract just cache verify # containment and complete disk accounting just cache prune [cache-id] # deterministic preview; defaults to all just cache prune <id> --apply --reason "why" just cache enforce <cache-id> --reason "why" just cache clean <cache-id> # explicit cold-clean preview just cache clean all --apply --reason "why" ``` `prune` follows ordinary age/count policy and only recovers to `warm_size_bytes` after usage crosses `max_size_bytes`. `enforce` is an applied preflight and fails if protected state prevents compliance. `clean` is the explicit cold operation; it still preserves active leases and protected generations. Cargo's ordinary retention selects whole compilation units under each `cargo_target_roots` entry: a unit's fingerprint, build output, libraries, dep-info, executables and verified signed copies share one metadata hash and are removed together, fingerprint first, least recently used first. Every gate prefix salts workspace units with its checkout path, so incremental-only retention let stale prefixes fill the stage until enforcement refused every run (issue #205). Paths Cargo does not name by unit, such as uplifted binaries, count toward capacity but are never selected. Retention is ordered by owner, not only by clock (issue #276). The workspace wrapper links each unit's fingerprint directory to the checkout that compiled it (`capsem-owner`, a symlink, so reading it moves no atime and measuring never counts it). A unit whose checkout is gone is reclaimed on every prune, whatever its age. Under pressure other checkouts' units go first, then shared or unrecorded ones (third-party crates, sccache hits, units from before the link existed), and the working set -- the calling checkout (`--policy-repository`, the bounded command's root) and the cache authority -- last. Plain LRU made the next build of the current head cold whenever other checkouts had compiled since. The copies themselves stay: the salt cannot go until Cargo freshness is content-based, and `-Zchecksum-freshness` is still unstable on the pinned toolchain. Native Cargo output locks protect the whole stage and are acquired again through deletion; even an explicit cold clean preserves their lock inodes. The Cargo maximum is held around compilation, not by any one step. Every exclusive gate command whose plan claims `workspace_binaries` holds `CargoCacheBound` (`gate/cachecontrol.py`), which enforces before the first step and after the last, failed runs included; `run-bounded-command.py` enforces before and after direct Cargo. Do not add an enforcement step to a fragment: a prerequisite of the host build left clippy, coverage and every other compile unbounded, and nothing enforced after any compile, until the target reached 237 GB against 180 GiB. The `objects` store (`object_store = true`) is retained the same way, one generation at a time: a component receipt together with the objects no other receipt names, receipt first. An object two receipts share belongs to neither and is collected once no receipt names it; an unreferenced object (a canonicalized package or release view, whose bytes live on in the hardlinked view) is its own generation. `componentcache.restore` treats a missing or corrupt object as a miss, never an error, and a hit moves the receipt's clock; never touch an object's times, which are its hardlinked outputs' times. Every reader and publisher holds `leases.shared_use(paths, "objects")`, so a prune never removes bytes in use. Before this the stage was `none`, grew past its cap, and failed every prune on the machine. A leased `ephemeral` stage (`test-temp`) is scratch, not retention: a generation lives exactly as long as a process holds its lease. The owner leases `run-<pid>` before creating it and, at exit, releases the lease and removes the run and its lease through `operations.reclaim_generation`, the same guarded, journaled path a prune takes. A run no process leases (its owner was killed) has no live owner, and `prune` reclaims it whatever its age or size; a lease file whose generation is gone is collected too, and never counts toward `maximum_count`. Removal takes each lease exclusively, so a reused pid that leased after the plan keeps its directory (`ApplyResult.busy`). Before this, a dead run aged like a retained generation below a 200 GiB maximum: one ~7 GB run per release precheck and 1,900 lock files filled the disk while `prune` offered 21 MB. That liveness does not depend on the authority, so such a stage is swept whole (`cache/namespaces.py`, issue #272): every other `sha256(authority)[:8]` namespace's runs and leases are entries keyed `<namespace>/<run>` under the same lease guard, and an emptied namespace is removed with `rmdir`, never recursively. Only a real directory with a namespace name is entered; a symlink, a pre-namespace `capsem-test-*` directory, or a file that is not a run or a lease is never touched. Prune once saw only its own namespace, and about 350 retired ones held 2.4 GB. The source-keyed Python stages (`python-pycache`, `python-pytest`) gain a generation for every Python-source state any checkout launches a gate or bounded command from, and nothing but a complete gate's package and install rails used to prune them: `python-pycache` reached 5.4 GB against 2 GiB. The launcher that creates a generation leases it and then enforces both stages (`pythonenv.bound_source_keyed`); a warm launch pays nothing. Retained cache lifetime ends only through these typed operations. Do not add consumer-boundary releases, post-test eviction hooks, or other subsystem lifecycle paths that bypass the owner's warm/max/age/count policy. Every applied mutation is exact, ownership-scoped, and journaled. Do not use broad `docker system prune`, direct Tart deletion, `rm` over cache roots, or backend-specific cache CLI commands. Foreign Docker/Tart resources are visible only as native totals and are never deleted by Capsem. Repository-generation reclaim is anchored by an exact current tag. Docker's bulk image listing can briefly lag a completed BuildKit import, so the Docker adapter verifies that exact tag directly and supplies a protected typed resource to the planner. Never turn this into an unanchored retry or allow a missing exact inspection to prune older generations. Prune and enforcement never remove an image a checkout's gate uses. The gate declares it: `reclaim-image --apply` records its `--keep` tag for the calling checkout, the host builder's require step reclaims around `:latest`, and the image backend records each profile's dependency image (`capsem_builder.cache.dockercurrent`, stored in `[control.docker] current_stage`). A record is live while its checkout exists and it is younger than the Docker runtime's `maximum_age_hours`; a newer tag for the same checkout and slot supersedes it. Scanning a runtime requires `paths=` so every planner sees live records as `current`. Newest-by-creation is not current: a BuildKit cache hit keeps the old timestamp, one repository holds a tag per profile, and two checkouts each have their own. On 2026-09-29 guessing it made enforcement evict the release proof's current kernel-dependency image and the host builder; each rebuild re-ran the slowest network phase. Pressure on current images is reported as a violation, never resolved by evicting them. Only an explicit `clean` may remove a current image. ## Test efficiency Start with the cheapest evidence that owns the change: ```bash just fast-test just focus-test <owner> ``` `just test <source-commit>` is the complete local proof a release requires. Its journal reuses an exact successful source identity, and its graph carries valid artifact frontiers instead of rebuilding them. Low-impact paths are routed by `[test_admission]`; forcing complete proof repeatedly is rate-limited by commit distance. Release commands consume its journal: they refuse a source without a passing run for the same commit or tree. After a failed or interrupted full attempt, automatic retries are refused. Use focused checks; obtain explicit approval for each reasoned full retry. The existing run journals supply this guard, not another history store. Working-tree gates derive their private prefix name from the exact source digest. Do not replace it with a random run ID: Cargo fingerprints contain absolute source paths, so random prefixes turn an unchanged repeat into a rebuild. The gate owns sccache as a scoped `CompilerCache` resource, exports `SCCACHE_BASEDIRS` (plural), uses client-side mode, and stops the server during resource teardown. Do not manage its daemon in shell. Cargo runs a workspace unit as `sccache <workspace wrapper> /abs/rustc ...`. sccache 0.17 drops that compiler argument only when it is spelled `rustc`, so it parsed the absolute path as a second input ("multiple input files") and ran every workspace unit uncached (issue #277). The rustc workspace wrapper therefore hands a `rustc` unit to `$RUSTC_WRAPPER` itself when that is sccache; `test_cargo_workspace_cache.py` proves the second compile is a hit. Watch `sccache --show-stats` for hits on workspace crates, not just a low non-cacheable count: the outer call still counts as non-cacheable. Gate commands and bounded builds compile with `CARGO_INCREMENTAL=0` (`[toolchain] cargo_incremental` in `config/gate.toml`, exported by `gatelaunch.contained_environment` and `cachetooling.compiler_environment`). That is the measured workload-specific reason: a prefix is named by its source digest and workspace units are salted by its path, so an incremental session is never read back, an agent worktree lives one session, and on 2026-09-29 the sessions were 75-83 GB of a 171-208 GB target. sccache also refuses incremental output. Keep the dev profile at `debug = "line-tables-only"` for the same reason: full DWARF was 77% of every test executable (`tests/citadel/test_hot_build_contract.py`). Keep the checkout-local `build.rustc-workspace-wrapper` configured in Cargo. Cargo includes its resolved path in workspace artifact keys, separating source trees while third-party dependencies stay shared. Without it, an older snapshot can be incorrectly considered fresh after a newer worktree compiled into the shared target, even with sccache disabled. Test both output correctness across trees and warm reuse on return; clearing the cache would hide this failure. `cargo clippy` forces `RUSTC_WORKSPACE_WRAPPER` to its own `clippy-driver`, one path for every checkout, so plain clippy output in the shared target loses that key and a checkout can take another's lint result as its own (a false green or red). The gate and `run-bounded-command.py` therefore run clippy as `cargo check` through the checkout-local `toolchain.clippy_workspace_wrapper` (`gate/clippyrun.py`). Never add a literal `cargo clippy` gate step; `build_system/tests/cache/test_clippy_checkout_key.py` refuses one and proves per-checkout verdicts on the pinned toolchain. Never make normal cache reuse opt-in. Do not clean caches to diagnose a product failure unless cold-state behavior is itself the subject under test. ## Changing policy 1. Run `just cache stats --json` and preserve the before inventory. 2. Change the owning cache entry in `config/cache.toml`. 3. Update Pydantic validation and focused cache tests together. 4. Run `just cache verify` and the cache, gate, and Citadel suites. 5. For a size change, run one complete gate with permissive maxima, record the post-run inventory, and derive warm/max values from observed retained work plus measured generation needs. Validate the ratchet with an exact-repeat run so it proves reuse rather than merely fitting a cold build. Do not infer a cache limit from filesystem capacity: retention decisions stay on owned usage. The maxima do answer to the disk as a set, through `[budget]` in `config/cache.toml` (`capsem_builder.cache.budget`). Every stage and runtime maximum together, plus `headroom_bytes`, must fit `filesystem_fraction` of the declared `minimum_filesystem_bytes` (checked when the policy loads) and of the real cache filesystem (checked by every exclusive gate command before it waits for the machine lock; an optional runtime that is not installed, such as Tart on Linux, counts zero; a GitHub-hosted runner is exempt). On 2026-09-29 the maxima summed to 839 GiB on a 484 GiB disk: every cache honoured its contract and the disk filled four times in two days. Raising one maximum means lowering another or raising the declared floor, deliberately. ## Debugging Read `cache/target/gate-runs/DIGEST.md` before reporting gate state. Then use: ```bash uv run --project build_system --frozen capsem-gate runs last --failed uv run --project build_system --frozen capsem-gate runs trend --step <label> just cache stats --json just cache contract <cache-id> ``` If runtime inventory fails, diagnose Docker/Colima or Tart through `/dev-setup`. Do not add fallback accounting or silently treat a required runtime as empty.