xerj-code

# XERJ.code — reference coding An agent that does not know an API guesses, runs, fails, and guesses again. Each lap costs **output** tokens, which are the expensive kind. This skill replaces laps with a lookup: clone the repositories that already contain a correct implementation, index them with XERJ, and retrieve the passage before writing. The trade is deliberate. Retrieval spends **input** tokens, which are cheaper per token and cacheable; loops spend **output** tokens. See [`COSTS.md`](../../docs/case-studies/reference-coding/COSTS.md) for the arithmetic and for the conditions under which this trade **loses**. > **Status, stated plainly: measured, and the value is real but narrow.** > (Full case study with real tokens and dollars: [`docs/case-studies/reference-coding/CASE_STUDY.md`](../../docs/case-studies/reference-coding/CASE_STUDY.md), 2026-08-05.) > > The comparison that matters is against **native Claude Code** — the same agent > with tools that would grep the source itself — not a bare model. Measured across > three regimes, same agent, one objective verdict (compiles + passes hidden tests): > > - **Unfamiliar code with a contract the model can't recall** (a seal, a > generational handle, a lazy refill, a specific hash scheme — measured across > **seven purpose-built libraries, 21 runs**): native and xerj both solve > **21/21**; **xerj uses 1.5× fewer output tokens and costs 1.3× less than > native**, because retrieval replaces the grep. Pure memory **fails 1/21** and > burns **6.5× the cost** flailing. This is the win, and it is decisive. The > compiler can leak an API *name* across a retry loop, but never a *contract*. > - **Memorised code** (any popular public crate — even a 256-value table, or a > protobuf-style varint): retrieval is **overhead**. Pure memory is cheapest; > injecting a big reference can be the *worst* arm, because the model re-emits > what it was handed. > > So: **use this when the reference is code the model has not memorised** — your > own private/proprietary code, an internal API, a post-cutoff or niche library. > On public library references the model already knows, it costs more than it > saves. The native-vs-xerj gap grows with corpus size (grep pulls the whole tree > into context; retrieval pulls one passage). Since [#977](https://github.com/xerj-org/xerj/issues/977) the whole loop is **inside the `xerj` binary** — `xerj corpus add` / `xerj corpus index` / `xerj code` — plus the `xerj_code_search` MCP tool for agents without a shell. The old wrapper scripts (`xc-corpus.sh`, `xc-index.sh`, `xc.py`) are gone; your existing `~/.xerj-code` corpora and indexes work unchanged (see [Migration](#migration-from-the-wrapper-scripts-977) below). ## Standing corpora for this repository (MANDATORY since 2026-08-06) In the xerj repo, retrieval-before-writing is **required** for non-trivial engine work, not optional. Each corpus below has a pinned, licence-reviewed definition in [`hub/`](hub/) — `xerj corpus add --from hub/<name>.json` rebuilds it at the same commits anyone else is using: | corpus | projects | use it for | |---|---|---| | `xerj-search` | lucene, tantivy, meilisearch, quickwit, sonic, elasticsearch | FTS, BM25, postings, merge policy, typo tolerance, segment layout, ES wire semantics | | `xerj-vector` | qdrant, usearch, instant-distance, hnswlib | HNSW build, neighbour heuristics, quantisation, filtered kNN | | `xerj-storage` | sled, fjall, redb | WAL, flush epochs, crash recovery, compaction, page allocation | | `xerj-columnar` | clickhouse | columnar storage, aggregation execution, compression codecs, vectorised scans | **Approach-only sources:** `elasticsearch` (AGPL-3.0 / SSPL-1.0 / Elastic-2.0) and `sonic` (MPL-2.0). Never copy their code into Apache-2.0 XERJ; for ES, pasting source would also falsify the project's public "shares no code with Elasticsearch" claim. `meilisearch` is MIT at the core but **BUSL-1.1 for its Enterprise Edition** parts — check the header/path first. Read the design, write our own implementation. Each manifest's `review.use` field says which bucket a repo is in; `hub/README.md` explains the three. ## Hub corpus or local corpus Two kinds of corpus sit on the same local node. Each answers a different kind of question: | | hub corpus | local corpus | |---|---|---| | what it holds | public material at a pinned commit: a library's source, a standard, a regulation, an advisory feed | files only this machine has: the project you are working in, private documents, exports | | use it when | you need a detail you cannot quote exactly from memory | the answer is in your own files | | add it | `xerj corpus add --from ./<slug>.json`, then `xerj corpus index <slug>` | `xerj autoindex <folder>` | | query it | `xerj code <slug> "<question>"` · MCP `xerj_code_search` | `xerj search "<words>"` · `xerj def "<symbol>"` · MCP `xerj_search` | Use neither for a fact you can already state exactly: on memorised content retrieval is overhead (see the status note above). A hub manifest comes from a checkout of the `corpus-hub` branch (`tools/xerj-code/hub/<slug>.json`) or, without a checkout, from its raw URL. Both `corpus index` and `code` talk to a running node: ```sh curl -fsSLO https://raw.githubusercontent.com/xerj-org/xerj/corpus-hub/tools/xerj-code/hub/<slug>.json xerj corpus add --from ./<slug>.json # last line: next: xerj corpus index <slug> xerj corpus index <slug> # ends: corpus '<slug>' searchable: <N> records xerj code <slug> "<question>" # passages headed by file path and line ``` `--from` reads a local file: v1.0.0-rc.95 does not fetch a URL, and passing one answers `no such manifest`. ## Choosing a corpus for the project at hand The four engine corpora above are this repository's standing set. The full live catalogue is browsable at **[hub.xerj.org](https://hub.xerj.org)**, with one line per corpus in [hub.xerj.org/llms.txt](https://hub.xerj.org/llms.txt); what a node has loaded is `xerj corpus list --url <node>`. Every corpus is pinned to a commit with its licence recorded per source. **G7** (a five-question retrieval gate whose suite was pre-registered *before* retrieval ran) has been graded for some corpora and not others, and each hub page says which. Pick by the project's domain, not by size: | the project is… | start with | why | |---|---|---| | a search / query engine | `xerj-search` | postings, BM25, merge policy, segment layout, ES wire semantics | | vector search / ANN | `xerj-vector`, `faiss`, `annoy` | HNSW build, neighbour heuristics, quantisation, filtered kNN; faiss is the reference ANN implementation, annoy the forest variant | | storage engine / KV | `xerj-storage`, `duckdb`, `rocksdb`, `leveldb`, `badger`, `lmdb`, `pebble`, `valkey-src` | WAL, crash recovery, compaction, page allocation; LSM-vs-B trees; pebble for L0 sub-levels and range-deletion semantics; valkey for an in-memory KV server's data structures | | analytical / columnar | `xerj-columnar`, `duckdb`, `arrow-rs`, `parquet-format`, `iceberg` | columnar layout, codecs, vectorised scans, table formats | | a database server | `sqlite-src`, `postgres-src` | parsers, planners, executors — the two most-studied codebases in the field | | messaging / consensus | `kafka-src`, `rabbitmq`, `nats-server`, `etcd-src`, `raft-rs`, `dragonboat` | log compaction, delivery semantics, quorum/lease correctness | | monitoring / timeseries | `influxdb`, `prometheus-src` | TSDB layout, downsampling, scrape/ingest paths | | TLS / crypto / tunnels | `rustls`, `openssl`, `s2n-tls`, `boringtun` | modern-TLS-by-hand, hardening precedent, constant-time patterns | | QUIC / HTTP / networking | `quinn`, `nghttp2`, `curl-src`, `libuv`, `caddy`, `envoy`, `nginx` | protocol state machines, connection pooling, event loops | | compression / serialization | `zstd`, `lz4`, `brotli`, `zlib-ng`, `snappy`, `protobuf`, `flatbuffers`, `capnproto`, `msgpack-c` | format design, blocked/entropy coding, zero-copy schemas | | secrets / config serving | `openbao` | KV store with seal/unseal, lease machinery | | API design review | `rust-api-guidelines`, `msft-api-guidelines`, `zalando-restful-api-guidelines`, `google-eng-practices` | naming, error shape, versioning; Zalando's is the most clause-complete REST policy set, google-eng-practices owns code-review doctrine | | protocol/spec conformance | `openapi-spec`, `graphql-spec`, `json-schema-spec`, `w3c-css`, `whatwg-specs`, `ecma262`, `nats-protocol`, `kafka-protocol` | the normative text an implementation is judged against — HTML/DOM/URL living standards, ECMAScript grammar, wire-protocol framing | | locating the right RFC before reading it | `ietf-rfc-abstracts` | one record per RFC (9,015): number, title, date, abstract — "which RFC actually covers X" without pulling full texts (TLP-5.0) | | production ops / SRE | `gitlab-runbooks`, `school-of-sre`, `tldr-pages`, `mozilla-ssl-configs`, `twelve-factor` | real incident playbooks, a full SRE course curriculum, command syntax by tool, deployed TLS configs, deployment doctrine | | administering Unix systems | `freebsd-handbook`, `openbsd-faq` | rc/ZFS/jail/networking procedures, OpenBSD's canonical FAQ positions | | documenting against a hosted API | `github-api-description` | the OpenAPI description of github.rest — field semantics, enum values, preview headers | | datastore operations docs | `valkey-docs`, `etcd-docs` | command semantics, config knobs, tuning guidance — the operational reference for each | | telemetry / observability wiring | the semconv half of `otel-proto` (candidate — G7 2/5, parked to the semantic lane) | OpenTelemetry semantic conventions for attribute names; lexical rounds could not bridge paraphrase-to-normative-prose | | internationalisation | `unicode-cldr` | CLDR locale data structure, calendar/number/date skeletons | | security review of Rust dependencies | `rustsec-advisories`, the `rust-vulns` pack (`xerj corpus add --from` the hub recipe) | one text record per advisory (1,963: RustSec + crates.io OSV, zero alias overlap): affected versions, fixed releases, affected-function symbol lists on 271; the pack stays the vehicle for patched-pair mining (#1111's defect-class study) | | security review of Python dependencies | `pysec` | PYSEC advisory records with affected and fixed versions, CVE/GHSA aliases | | cross-ecosystem dependency audit | `ghsa-db` | GitHub-reviewed advisories (36,263 text records): CVSS vectors, patched ranges, affected packages across npm/PyPI/Go/Rust/maven and the rest | | security review of Go modules | `go-vulndb` | Go-team-curated advisories (4,577): affected packages and symbols, fixed versions, GO ids with CVE aliases | | regex denial-of-service precedent | `redos-precedent` | 634 regex-specific ReDoS advisories (CWE-1337/185 class, not bare CWE-400): the pathological input, the affected function, the bounded rewrite | | crypto misuse precedent | `crypto-misuse-precedent` | 721 CWE-326/327/328/329/347-class advisories: weak ciphers, hardcoded keys, nonce reuse, signature-verification bypass, padding oracles, ECB | | vulnerability / weakness taxonomy | `mitre-cwe`, `mitre-capec` | CWE weakness definitions and CAPEC attack patterns, id-level | | adversary techniques / detection engineering | `mitre-attck` | ATT&CK v19.2 per-technique records: technique ids, tactics, detection data sources, procedure examples (enterprise/mobile/ics; software and group objects out of scope) | | patch-priority triage | `cisa-kev` | the Known Exploited Vulnerabilities catalogue, one record per CVE — exploitation status in the wild, binding remediation dates | | web application security how-to | `owasp-cheatsheets` | the Cheat Sheet Series' countermeasure recipes per topic — session management, authentication secrets, error handling, upload handling | | planning a web app penetration test | `owasp-wstg` | the Web Security Testing Guide v4.2 catalogue: per-test objectives and method, organised by category (authn, authz, session, input handling) | | triaging an active campaign | `cisa-alerts` | CISA advisory IOCs, required actions and mitigations (2025-05..2026-10 window) | | querying endpoint visibility | `osquery-config-examples` | real deployed osquery packs and scheduled-query shapes | | non-software (tax, employment, UK law) | `ecfr-title-12`, `ecfr-title-29`, `ecfr-title-26`, `uk-legislation` | consolidated current regulation, section-per-file | | aviation (FARs) | `ecfr-title-14` | 14 CFR section-per-file: visibility minimums, transponder airspace, inspection intervals, right-of-way | | maritime rules of the road | `navrules` | COMDTINST M16672.2D per-rule text — risk of collision, narrow channels, crossing situations, lights and sound signals | | accessible UI components | `govuk-design-system` | GOV.UK component options and accessibility criteria — the exact option semantics (maxwords vs maxlength, divider text, focus-on-load) | | pharma / drug labels | `dailymed` | FDA label sections: indications, contraindications, administration timing | | clinical treatment guidance | `cdc-clinical` | CDC STI treatment regimens and doses, syphilis desensitization, measles clinical signs | | food-service safety | `fda-food-code` | FDA Food Code 2022 provisions: hand-contact surfaces, Time as a Public Health Control, date marking, bare-hand readiness | | traffic control devices | `mutcd` | FHWA MUTCD 11th Ed (Rev 1): sign and signal warrants, markings, work-zone typical applications | | building works (England) | `uk-building-regs` | Building Regulations 2010 (SI 2010/2214) per-provision: Parts A–P requirements, Wales variants labelled, 44ZB/44ZC dual numbering documented | | EU digital regulation | `eurlex-core` | GDPR, AI Act, DSA, DMA, NIS2 per-article and per-recital from the EUR-Lex Cellar (annexes excluded by design) | | accessibility (built environment) | `ada-2010` | ADA 2010 Standards: reach ranges, clear floor space, slope ratios, per-section | **Inside the security block, pick twice — ecosystem first, defect class second.** Advisories about the same CVE exist in several databases; start with the corpus whose ecosystem matches the project (`rustsec-advisories` for Rust crates, `pysec` for PyPI, `go-vulndb` for Go modules, `ghsa-db` when the tree is mixed or the ecosystem is unclear — it is the superset, GitHub-reviewed). Then, when the review is about a *pattern you are looking at* rather than a package — a regex fed user input, a cipher or key-management call site — switch to the stratum corpus (`redos-precedent`, `crypto-misuse-precedent`): smaller, selected for exactly that defect class, so the mechanism (the pathological input, the weak primitive, the bounded rewrite that fixed it) ranks instead of thousands of generic advisories. Three rules that keep the choice honest: 1. **A corpus only helps on code you have not memorised.** On popular public library code the model already knows, retrieval is overhead — the measured crossover is in `measure/DEVQA_RESULTS.md`. Private, internal, niche and post-cutoff code is where it wins decisively (bare-P 20% vs with-corpus-X 98% across 33 measured corpora — `benchmarks/corpus-tasks/IMPACT.md`). 2. **Shape beats size.** A corpus whose answer sits in one giant file (a 3 MB single-page spec, a 50 MB JSON bundle) loses to a section-per-file mirror of the same content — whole-file indexing skips what it cannot chunk, and near-duplicate bundles crowd the answer. The hub's G7 gate exists to catch exactly this; corpora that failed it were demoted with the fix pre-registered, not padded into "live". 3. **Cite what you adapted.** A retrieved hit is evidence of how *that* project solved it under *its* licence and constraints — `file:line` in the comment, and check the manifest's `review.use` bucket before copying anything. ### Operational traps, both hit in practice - **Nothing under `/tmp`.** Clones live in `~/.xerj-code/corpora/`, the index in `~/.xerj-code/data` (override the root with `XERJ_CODE_HOME`). A `/tmp` data dir is lost on reboot and the whole corpus silently retrieves nothing. - **`xerj corpus index <corpus> --fresh` is the rebuild, and it never costs you a working index.** It builds a replacement *beside* the existing one — under `xc-<corpus>-b<stamp>-*`, with an autoindex `--state-dir` of its own in `~/.xerj-code/autoindex-state/<corpus>/` — verifies it (autoindex exited `0` or `3` **and** `_count > 0`), switches `state/<corpus>.json` to it by atomic rename, and only then deletes the old indices, by exact name. A build that fails or comes back empty is removed and the old index, the old state file and the old state directory stay exactly as they were. Use it after the data dir was wiped or moved, when `xerj code` says the index is older than 30 days, or when a plain re-run says the state directory "cannot become generation authority". It is **not** `xerj autoindex --fresh`, which the command never forwards: that flag only discards autoindex's resume journal and is refused outright once a corpus generation has committed — which is why `--fresh` used to fail on every corpus that had been indexed before