xerj-code

# XERJ.code — reference coding An agent that does not know an API guesses, runs, fails, and guesses again. Each lap costs **output** tokens, which are the expensive kind. This skill replaces laps with a lookup: clone the repositories that already contain a correct implementation, index them with XERJ, and retrieve the passage before writing. The trade is deliberate. Retrieval spends **input** tokens, which are cheaper per token and cacheable; loops spend **output** tokens. See [`COSTS.md`](../../docs/case-studies/reference-coding/COSTS.md) for the arithmetic and for the conditions under which this trade **loses**. > **Status, stated plainly: measured, and the value is real but narrow.** > (Full case study with real tokens and dollars: [`docs/case-studies/reference-coding/CASE_STUDY.md`](../../docs/case-studies/reference-coding/CASE_STUDY.md), 2026-08-05.) > > The comparison that matters is against **native Claude Code** — the same agent > with tools that would grep the source itself — not a bare model. Measured across > three regimes, same agent, one objective verdict (compiles + passes hidden tests): > > - **Unfamiliar code with a contract the model can't recall** (a seal, a > generational handle, a lazy refill, a specific hash scheme — measured across > **seven purpose-built libraries, 21 runs**): native and xerj both solve > **21/21**; **xerj uses 1.5× fewer output tokens and costs 1.3× less than > native**, because retrieval replaces the grep. Pure memory **fails 1/21** and > burns **6.5× the cost** flailing. This is the win, and it is decisive. The > compiler can leak an API *name* across a retry loop, but never a *contract*. > - **Memorised code** (any popular public crate — even a 256-value table, or a > protobuf-style varint): retrieval is **overhead**. Pure memory is cheapest; > injecting a big reference can be the *worst* arm, because the model re-emits > what it was handed. > > So: **use this when the reference is code the model has not memorised** — your > own private/proprietary code, an internal API, a post-cutoff or niche library. > On public library references the model already knows, it costs more than it > saves. The native-vs-xerj gap grows with corpus size (grep pulls the whole tree > into context; retrieval pulls one passage). Since [#977](https://github.com/xerj-org/xerj/issues/977) the whole loop is **inside the `xerj` binary** — `xerj corpus add` / `xerj corpus index` / `xerj code` — plus the `xerj_code_search` MCP tool for agents without a shell. The old wrapper scripts (`xc-corpus.sh`, `xc-index.sh`, `xc.py`) are gone; your existing `~/.xerj-code` corpora and indexes work unchanged (see [Migration](#migration-from-the-wrapper-scripts-977) below). ## Standing corpora for this repository (MANDATORY since 2026-08-06) In the xerj repo, retrieval-before-writing is **required** for non-trivial engine work, not optional. Each corpus below has a pinned, licence-reviewed definition in [`hub/`](hub/) — `xerj corpus add --from hub/<name>.json` rebuilds it at the same commits anyone else is using: | corpus | projects | use it for | |---|---|---| | `xerj-search` | lucene, tantivy, meilisearch, quickwit, sonic, elasticsearch | FTS, BM25, postings, merge policy, typo tolerance, segment layout, ES wire semantics | | `xerj-vector` | qdrant, usearch, instant-distance, hnswlib | HNSW build, neighbour heuristics, quantisation, filtered kNN | | `xerj-storage` | sled, fjall, redb | WAL, flush epochs, crash recovery, compaction, page allocation | | `xerj-columnar` | clickhouse | columnar storage, aggregation execution, compression codecs, vectorised scans | **Approach-only sources:** `elasticsearch` (AGPL-3.0 / SSPL-1.0 / Elastic-2.0) and `sonic` (MPL-2.0). Never copy their code into Apache-2.0 XERJ; for ES, pasting source would also falsify the project's public "shares no code with Elasticsearch" claim. `meilisearch` is MIT at the core but **BUSL-1.1 for its Enterprise Edition** parts — check the header/path first. Read the design, write our own implementation. Each manifest's `review.use` field says which bucket a repo is in; `hub/README.md` explains the three. ## Choosing a corpus for the project at hand The four engine corpora above are this repository's standing set. The full live catalogue — every corpus pinned, licence-reviewed and **G7-graded** (a five-question retrieval gate whose suite was pre-registered *before* retrieval ran) — is browsable at **[hub.xerj.org](https://hub.xerj.org)** and queryable live with `xerj corpus list --url <node>`. Pick by the project's domain, not by size: | the project is… | start with | why | |---|---|---| | a search / query engine | `xerj-search` | postings, BM25, merge policy, segment layout, ES wire semantics | | vector search / ANN | `xerj-vector`, `faiss`, `annoy` | HNSW build, neighbour heuristics, quantisation, filtered kNN; faiss is the reference ANN implementation, annoy the forest variant | | storage engine / KV | `xerj-storage`, `duckdb`, `rocksdb`, `leveldb`, `badger`, `lmdb`, `pebble`, `valkey-src` | WAL, crash recovery, compaction, page allocation; LSM-vs-B trees; pebble for L0 sub-levels and range-deletion semantics; valkey for an in-memory KV server's data structures | | analytical / columnar | `xerj-columnar`, `duckdb`, `arrow-rs`, `parquet-format`, `iceberg` | columnar layout, codecs, vectorised scans, table formats | | a database server | `sqlite-src`, `postgres-src` | parsers, planners, executors — the two most-studied codebases in the field | | messaging / consensus | `kafka-src`, `rabbitmq`, `nats-server`, `etcd-src`, `raft-rs`, `dragonboat` | log compaction, delivery semantics, quorum/lease correctness | | monitoring / timeseries | `influxdb`, `prometheus-src` | TSDB layout, downsampling, scrape/ingest paths | | TLS / crypto / tunnels | `rustls`, `openssl`, `s2n-tls`, `boringtun` | modern-TLS-by-hand, hardening precedent, constant-time patterns | | QUIC / HTTP / networking | `quinn`, `nghttp2`, `curl-src`, `libuv`, `caddy`, `envoy`, `nginx` | protocol state machines, connection pooling, event loops | | compression / serialization | `zstd`, `lz4`, `brotli`, `zlib-ng`, `snappy`, `protobuf`, `flatbuffers`, `capnproto`, `msgpack-c` | format design, blocked/entropy coding, zero-copy schemas | | secrets / config serving | `openbao` | KV store with seal/unseal, lease machinery | | API design review | `rust-api-guidelines`, `msft-api-guidelines`, `zalando-restful-api-guidelines`, `google-eng-practices` | naming, error shape, versioning; Zalando's is the most clause-complete REST policy set, google-eng-practices owns code-review doctrine | | protocol/spec conformance | `openapi-spec`, `graphql-spec`, `json-schema-spec`, `w3c-css`, `whatwg-specs`, `ecma262`, `nats-protocol`, `kafka-protocol` | the normative text an implementation is judged against — HTML/DOM/URL living standards, ECMAScript grammar, wire-protocol framing | | locating the right RFC before reading it | `ietf-rfc-abstracts` | one record per RFC (9,015): number, title, date, abstract — "which RFC actually covers X" without pulling full texts (TLP-5.0) | | production ops / SRE | `gitlab-runbooks`, `school-of-sre`, `tldr-pages`, `mozilla-ssl-configs`, `twelve-factor` | real incident playbooks, a full SRE course curriculum, command syntax by tool, deployed TLS configs, deployment doctrine | | administering Unix systems | `freebsd-handbook`, `openbsd-faq` | rc/ZFS/jail/networking procedures, OpenBSD's canonical FAQ positions | | documenting against a hosted API | `github-api-description` | the OpenAPI description of github.rest — field semantics, enum values, preview headers | | datastore operations docs | `valkey-docs`, `etcd-docs` | command semantics, config knobs, tuning guidance — the operational reference for each | | telemetry / observability wiring | the semconv half of `otel-proto` (candidate — G7 2/5, parked to the semantic lane) | OpenTelemetry semantic conventions for attribute names; lexical rounds could not bridge paraphrase-to-normative-prose | | internationalisation | `unicode-cldr` | CLDR locale data structure, calendar/number/date skeletons | | security review of Rust dependencies | `rustsec-advisories`, the `rust-vulns` pack (`xerj corpus add --from` the hub recipe) | one text record per advisory (1,963: RustSec + crates.io OSV, zero alias overlap): affected versions, fixed releases, affected-function symbol lists on 271; the pack stays the vehicle for patched-pair mining (#1111's defect-class study) | | security review of Python dependencies | `pysec` | PYSEC advisory records with affected and fixed versions, CVE/GHSA aliases | | cross-ecosystem dependency audit | `ghsa-db` | GitHub-reviewed advisories (36,263 text records): CVSS vectors, patched ranges, affected packages across npm/PyPI/Go/Rust/maven and the rest | | security review of Go modules | `go-vulndb` | Go-team-curated advisories (4,577): affected packages and symbols, fixed versions, GO ids with CVE aliases | | regex denial-of-service precedent | `redos-precedent` | 634 regex-specific ReDoS advisories (CWE-1337/185 class, not bare CWE-400): the pathological input, the affected function, the bounded rewrite | | crypto misuse precedent | `crypto-misuse-precedent` | 721 CWE-326/327/328/329/347-class advisories: weak ciphers, hardcoded keys, nonce reuse, signature-verification bypass, padding oracles, ECB | | vulnerability / weakness taxonomy | `mitre-cwe`, `mitre-capec` | CWE weakness definitions and CAPEC attack patterns, id-level | | adversary techniques / detection engineering | `mitre-attck` | ATT&CK v19.2 per-technique records: technique ids, tactics, detection data sources, procedure examples (enterprise/mobile/ics; software and group objects out of scope) | | patch-priority triage | `cisa-kev` | the Known Exploited Vulnerabilities catalogue, one record per CVE — exploitation status in the wild, binding remediation dates | | web application security how-to | `owasp-cheatsheets` | the Cheat Sheet Series' countermeasure recipes per topic — session management, authentication secrets, error handling, upload handling | | planning a web app penetration test | `owasp-wstg` | the Web Security Testing Guide v4.2 catalogue: per-test objectives and method, organised by category (authn, authz, session, input handling) | | triaging an active campaign | `cisa-alerts` | CISA advisory IOCs, required actions and mitigations (2025-05..2026-10 window) | | querying endpoint visibility | `osquery-config-examples` | real deployed osquery packs and scheduled-query shapes | | non-software (tax, employment, UK law) | `ecfr-title-12`, `ecfr-title-29`, `ecfr-title-26`, `uk-legislation` | consolidated current regulation, section-per-file | | aviation (FARs) | `ecfr-title-14` | 14 CFR section-per-file: visibility minimums, transponder airspace, inspection intervals, right-of-way | | maritime rules of the road | `navrules` | COMDTINST M16672.2D per-rule text — risk of collision, narrow channels, crossing situations, lights and sound signals | | accessible UI components | `govuk-design-system` | GOV.UK component options and accessibility criteria — the exact option semantics (maxwords vs maxlength, divider text, focus-on-load) | | pharma / drug labels | `dailymed` | FDA label sections: indications, contraindications, administration timing | | clinical treatment guidance | `cdc-clinical` | CDC STI treatment regimens and doses, syphilis desensitization, measles clinical signs | | food-service safety | `fda-food-code` | FDA Food Code 2022 provisions: hand-contact surfaces, Time as a Public Health Control, date marking, bare-hand readiness | | traffic control devices | `mutcd` | FHWA MUTCD 11th Ed (Rev 1): sign and signal warrants, markings, work-zone typical applications | | building works (England) | `uk-building-regs` | Building Regulations 2010 (SI 2010/2214) per-provision: Parts A–P requirements, Wales variants labelled, 44ZB/44ZC dual numbering documented | | EU digital regulation | `eurlex-core` | GDPR, AI Act, DSA, DMA, NIS2 per-article and per-recital from the EUR-Lex Cellar (annexes excluded by design) | | accessibility (built environment) | `ada-2010` | ADA 2010 Standards: reach ranges, clear floor space, slope ratios, per-section | **Inside the security block, pick twice — ecosystem first, defect class second.** Advisories about the same CVE exist in several databases; start with the corpus whose ecosystem matches the project (`rustsec-advisories` for Rust crates, `pysec` for PyPI, `go-vulndb` for Go modules, `ghsa-db` when the tree is mixed or the ecosystem is unclear — it is the superset, GitHub-reviewed). Then, when the review is about a *pattern you are looking at* rather than a package — a regex fed user input, a cipher or key-management call site — switch to the stratum corpus (`redos-precedent`, `crypto-misuse-precedent`): smaller, selected for exactly that defect class, so the mechanism (the pathological input, the weak primitive, the bounded rewrite that fixed it) ranks instead of thousands of generic advisories. Three rules that keep the choice honest: 1. **A corpus only helps on code you have not memorised.** On popular public library code the model already knows, retrieval is overhead — the measured crossover is in `measure/DEVQA_RESULTS.md`. Private, internal, niche and post-cutoff code is where it wins decisively (bare-P 20% vs with-corpus-X 98% across 33 measured corpora — `benchmarks/corpus-tasks/IMPACT.md`). 2. **Shape beats size.** A corpus whose answer sits in one giant file (a 3 MB single-page spec, a 50 MB JSON bundle) loses to a section-per-file mirror of the same content — whole-file indexing skips what it cannot chunk, and near-duplicate bundles crowd the answer. The hub's G7 gate exists to catch exactly this; corpora that failed it were demoted with the fix pre-registered, not padded into "live". 3. **Cite what you adapted.** A retrieved hit is evidence of how *that* project solved it under *its* licence and constraints — `file:line` in the comment, and check the manifest's `review.use` bucket before copying anything. ### Operational traps, both hit in practice - **Nothing under `/tmp`.** Clones live in `~/.xerj-code/corpora/`, the index in `~/.xerj-code/data` (override the root with `XERJ_CODE_HOME`). A `/tmp` data dir is lost on reboot and the whole corpus silently retrieves nothing. - **`xerj corpus index <corpus> --fresh` is the rebuild, and it never costs you a working index.** It builds a replacement *beside* the existing one — under `xc-<corpus>-b<stamp>-*`, with an autoindex `--state-dir` of its own in `~/.xerj-code/autoindex-state/<corpus>/` — verifies it (autoindex exited `0` or `3` **and** `_count > 0`), switches `state/<corpus>.json` to it by atomic rename, and only then deletes the old indices, by exact name. A build that fails or comes back empty is removed and the old index, the old state file and the old state directory stay exactly as they were. Use it after the data dir was wiped or moved, when `xerj code` says the index is older than 30 days, or when a plain re-run says the state directory "cannot become generation authority". It is **not** `xerj autoindex --fresh`, which the command never forwards: that flag only discards autoindex's resume journal and is refused outright once a corpus generation has committed — which is why `--fresh` used to fail on every corpus that had been indexed before ([#930](https://github.com/xerj-org/xerj/issues/930)). - **A record count the node does not answer is never read as zero.** The count is a tri-state: a number, a 404 (no index matches — genuinely zero), or *Unknown* (timeout, 5xx, unparseable reply). *Unknown* authorises no delete and no swap: if the node cannot count the existing index, it is presumed to be a working one and no failed build is kept over it; if it cannot count the new build, nothing is deleted and nothing is switched. Only a number — or a 404 — authorises a delete or a swap. - **An interrupted FIRST build is kept, and `xerj code` says so.** When a build fails after writing records and there is no working index to fall back to, the command keeps it rather than leave no corpus at all, and records `salvaged: true` plus the real `autoindex_exit` in the state file. Such a build can be partial, so `xerj code` warns on stderr with every query that coverage is INCOMPLETE — **a miss is then not evidence that the code is absent** — and `xerj corpus list` marks the corpus. A plain `xerj corpus index <corpus>` resumes it (same prefix, same state directory) and clears the mark when it finishes. - **A plain `xerj corpus index <corpus>` updates in place.** It re-runs autoindex against the recorded build's prefix and state directory, so additions, edits, deletions and renames reconcile incrementally. - **The state file has two prefixes on purpose.** `prefix` is always `xc-<corpus>` — the whole namespace, so anything that globs `xc-<corpus>*` keeps working across rebuilds. `index_prefix` is the one verified build, and it is what `xerj code` queries, so a half-built replacement never leaks into answers during a rebuild. - **During a rebuild both builds exist**, so the node briefly holds the corpus twice. Budget disk for it on a large corpus. - Always verify: `curl -s "$URL/xc-<corpus>*/_count"` must be > 0. The command does this itself and says so; do it again if you are scripting around it. ## When this is worth it Use it when the task touches code someone else has already written correctly: - an unfamiliar library, framework, or protocol - porting an algorithm whose reference implementation exists - matching an established convention across a large codebase - **any time you have already looped twice on the same error** — that is the signal that guessing is not converging Do not use it for code that only exists in this repository (ordinary file reads are cheaper), for one-line edits, or when you already know the API. Indexing has a fixed setup cost; a task you would finish in two tool calls will not repay it. ## The loop ``` xerj corpus add <name> <git-url>... # clone reference repos (once per domain)