ironroot-resolver

# Ironroot resolver Recursive DNSSEC-validating resolver. Workspace `ironroot/`: `ironroot` (userspace), `ironroot-common` (no_std shared parser), `ironroot-ebpf` (XDP L0). Spec `IDEA.md.txt`. ## When to use Anything touching this resolver: the pipeline, the parsers, the caches, the DNSSEC validator, or the XDP fast path. Read the trap sections before writing a test -- most of the cost here is a wrong assumption about an API, not a hard bug. ## Running it - Bind flag is `--listen <SocketAddr>`, NOT `--bind`. Default `127.0.0.1:5353` collides with Windows mDNS, so use a high port (e.g. `--listen 127.0.0.1:15410`). - `--no-validate` exists only with `--features diagnostics-no-validate`. Without it the field is not compiled in, so a production binary has no bypass to invoke. - `--trust-anchor <path>` points at the RFC 5011 anchor file. Differential suite (network-dependent): IRONROOT_UNDER_TEST=127.0.0.1:15411 IRONROOT_REFERENCE=8.8.8.8 \ cargo test -p ironroot --test differential -- --ignored --test-threads=1 ## Tests and gates | Suite | Count | Network | |---|---|---| | unit | 106 | no | | conformance | 49 | no | | properties (proptest) | 16 | no | | concurrency | 9 | partly | | hermetic e2e | 4 + 10 ignored | no | | common | 39 | no | | differential | 9 | yes | `scripts/verify.sh` runs 11 gates, logs each to `logs/`, prints a summary. `--quick`, `--only <gate>`, `--list`. Treats a compiler *warning* as a failure: `#![allow(dead_code)]` hid a real bug here. `scripts/diagnose.{sh,py}` queries a live resolver and prints a full decode -- flags, counts, a hand-rolled record walk reporting where a bad message stopped parsing, and a TCP second opinion. ## Remote Private repo: https://github.com/LoopyLuci/ironroot -- branch `main`, tag `v0.1.0`. Four CI jobs: userspace tests, hermetic e2e, the validation-bypass binary grep, and the TLC model check. The bypass grep is mutation-tested: it fails against a `diagnostics-no-validate` build and passes against a secure one. `tla2tools.jar` is fetched by `scripts/fetch-tla.sh`, not committed. ## Measure the running resolver; do not read it Reading the code found nothing. Timing it found three bugs in one pass: - `h0.debian.org` took **14 s**, with a visible 10 s gap between two hops. The per-attempt timeout doubled per retry (RFC 1035 §4.2.2) with no total cap, so one dead server cost 1.5 + 3 + 6 s. Now bounded by `max_total_per_server`. - `--upstream-timeout-ms` reached only the validator, so it had no effect on the iterator -- the part that waits most. It now lives on `UpstreamConfig`. - UDP datagrams were **dropped by the kernel** under burst, because the socket used the OS default buffer. A dropped query is indistinguishable from being down. Buffers now requested at 4 MiB; the kernel grants 8. Probed and found correct, so left alone: TCP, 14 hostile datagrams, flag hygiene across TC/opcode/AA/CD/RD/Z, and the rate limiter (207/193 at defaults). ### Two measurement errors that first looked like resolver bugs Both are worth remembering because each reads as a server fault: - **Firing 400 datagrams in a tight loop tests the socket buffer, not the limiter.** The tokio task never gets scheduled, so the buffer absorbs them and the limiter sees nothing. Pace the burst or yield between sends. - **A UDP socket reused after a timeout reports a spurious ICMP reset on Windows.** Zero resets on a fresh socket for the same queries. Use one socket per query when measuring. ### `#[cfg(unix)]` code that only ever ran on Linux Socket buffer tuning took three attempts that all compiled and passed on Windows while panicking every runtime worker on Linux, because the block was `cfg(unix)` and never executed there. `UdpSocket::from_std` registers a *blocking* socket, which tokio permits only where blocking is legal, and an `async fn` body never is -- on any platform. Tuning belongs in `main`, before the runtime exists, where it is merely a syscall. The general rule: anything behind a platform cfg has never been executed unless it was run on that platform. `cargo build` on Windows is not a test of it. - **DO bit read from the OPT rclass** instead of bit 15 of the OPT TTL (RFC 6891 §6.1.2). The XDP path did this, so the L0 key never matched a DO-bearing client: a silent 100% fast-path miss. Userspace had it right; pin the two together. - **Rate limiter after the cache.** The limiter is the pipeline's first stage, but the L1 lookup ran *before* the pipeline, so cache hits skipped it and a client asking for one popular cached name got unlimited service. `Kernel` now holds the limiter and checks it before the cache. - **`cacheable` accepted TTL 0** (RFC 2181 §5.2). Storing it is a correctness bug. - **Referrals were cacheable and read as negative answers.** NOERROR-with-no-answers is both a referral's shape and NODATA's. Now `cacheable` requires `ancount > 0` and `is_negative` requires an SOA (RFC 2308 §2.1). - **The validator built a fresh transport per request**, discarding `--upstream-timeout-ms`. Found only after the lib/bin split removed `#![allow(dead_code)]`. ## Sweep the corpus; don't assert a hand-written list `ironroot/src/bin/differential-corpus.rs` compares 121 questions against a reference and reports **every** disagreement. Run it before concluding a resolver behaves. Five comparison traps, each producing false positives before being fixed: - **Never compare AD with the reference.** A validating resolver is *required* to set AD (RFC 4035 3.2.3) and public resolvers do not echo it. Only "never AD on a failure" is a real invariant. - **Record contents are not comparable.** Anycast and round-robin return a different address subset per query -- `reddit.com` gave five addresses at the reference, two here, both correct. Compare **names and types**. - **The CNAME chain below the queried name may differ.** The reference chased Fastly, we chased Akamai. Compare only records owned by the queried name. - **A cold cache exceeds the per-query timeout.** Retry once; report as latency, not correctness. - **A public reference throttles a fast loop.** Pace it, and stop after three consecutive reference misses instead of reporting the rest as failures. ## The harness must not measure itself Three separate defects all had the same shape: the gate or sweep reported its own limits as resolver failures, and each read convincingly like a resolver bug. - **The corpus sweep tripped the resolver's own rate limiter.** 121 questions in quick succession from one client address exceed the default 100 qps / 200 burst, and the rejections were counted as timeouts: 66/121 agreed, with 54 "no bytes back". The resolver was throttling the test, not failing. Raise the limits for the harness resolver only; production defaults are correct and stay tested. - **The sweep's 8s per-query timeout was measuring its own pacing**, not resolution time. Four names timed out that resolve in 0.1-1.3s standalone. At 20s: 121/121. - **`cargo clean` never ran** because a bare `cargo` does not exist on this host, only `cargo.exe`, so the gate meant to prevent inspecting a stale binary inspected a stale binary. Before believing a "resolver failure", ask what the harness was doing when it happened. A timeout and a rejection are both harness-shaped. ## A Windows socket error is not a fatal one `recv_from(&mut buf).await?` in the UDP accept loop removed the listener on the first recoverable error. Windows reports `WSAECONNRESET` (10054) on `recv_from` after an ICMP port-unreachable from an earlier send -- routine, and caused by any client that vanishes between our reply and its retransmit. The symptom is deceptive: the process is alive and still holds the socket, so health checks pass, but nothing is answered any more. It showed up as the corpus sweep agreeing on 80 of 121 questions and then reporting "no bytes back" for every remainder -- which reads as rate limiting, not a dead listener. Count transient errors and continue; log the first and every hundredth so a persistent fault stays visible. Give up on a bound number of consecutive errors rather than spinning forever. `ConnectionReset`, `Interrupted`, `WouldBlock` and `TimedOut` are all recoverable; anything else is a genuine fault. ## BPF: the verifier tests the object, not the loader `verify-bpf.sh` rebuilds the BPF *object* from the Windows tree every run. The *loader* that actually executes it lives inside the initramfs at `/home/ironroot/vm/root_base/loader` and is **not** rebuilt. Days of edits to `vm/load.c` were silently untested, and the gate kept reporting the same error the whole time. After touching `load.c`: ``` cp vm/load.c ~/vm/ && cd ~/vm && gcc -O2 -static -o load load.c && cp load root_base/loader ``` Three bugs lived in that untested file. A `call` immediate is a **signed instruction distance from the calling instruction**, not a byte offset and not a file offset — the old code wrote `symbol value + section file offset`, which is short by the program's own start offset and puts `.text.unlikely.` outside the program entirely ("call to invalid destination"). Code in another section must be **copied into** the program array, since a BPF program is one contiguous block. And `insn_cnt` has to count the appended code, or the destinations exist outside the submitted range and produce the *same* error — which is what made the second bug hard to spot. Always print the value actually submitted, not the variable that happens to be nearby. A diagnostic reading a stale variable is worse than none: it reports a plausible number and hides the effect of the change. ## Never test against a stale binary A temporary debug line failed to compile, so the "fix worked" run actually exercised the previous artifact and reported a pass. `cargo build` returning non-zero is not an optional detail: check the exit status before believing any result from a rebuilt binary, and prefer a mutation that must *fail* to a test that must pass -- a green test that never ran proves nothing. The same trap when a mutation silently does not apply: assert the mutated text differs from the original before running anything, or the "detected" result is fiction. ## Gate wiring: a gate that cannot run must say so The corpus sweep was added to the `GATES` array but never dispatched. It printed `unknown gate: corpus` and the run still reported "all gates passed". A gate that cannot run must fail, not skip quietly -- a silent skip is a green tick it did not earn. Related, all found the hard way: `A=x B=y` on one line is a single command taking `A` as its name, so **neither** variable is set. `--bin foo` builds only `foo`, leaving sibling binaries absent so later checks run against a missing file -- and a missing executable reads as a *passing* security check. `[ -f "X:/..." ]` is always false from a POSIX shell, so Windows-form paths silently miss files that exist; and guessing the POSIX form (`/X/...` rather than MSYS's `/mnt/x/...`) is worse than not converting, because an unverified guess looks like a real path. ## DNSSEC verdicts: absence of evidence is not forgery The two worst bugs were verdict-mapping errors, in **different code paths**: | Situation | Correct | Was | |---|---|---| | Unsigned zone, answer has no RRSIG | Insecure | `Err` -> SERVFAIL | | Unsigned zone, NODATA with no SOA | Insecure | `Ok(Some(Bogus))` -> SERVFAIL | `Missing SOA record for NODATA` is only a violation in a **signed** zone. In an unsigned zone the server is not obliged to send one. `github.com` publishes no DS, so both its plain answers and its NODATA answers were refused -- one of the most-visited names on the Internet. The guard stays narrow: **zero RRSIG records anywhere** means Insecure. Anything carrying signatures that fails to parse still fails closed. Fix one arm and check the other -- the first fix did not cover the second. **Mutation-test every verdict fix.** Reverting the Bogus-path fix left the suite green until unsigned-NODATA coverage existed. ### Distinguish a broken zone from a resolver bug A sweep flagged `go.dns.pl` as SERVFAIL. Chasing it: `dns.pl` publishes a DS but serves no DNSKEY -- an unsigned island -- and Google and Cloudflare SERVFAIL on it too. Ironroot answers `dns.pl` with **AD=1**, more strictly than either. Broken zone, not a bug; SERVFAIL was right. Check whether independent references fail the same way before "fixing" anything. ## Two gaps found by probing the header, not the code Reading the resolver found nothing in these two places; a sweep of header fields did. Both produced a **correct-looking response with the AD bit set**, which is the worst kind of bug for a validating resolver: the answer looked verified. **No opcode validation existed at all.** Every opcode was resolved as QUERY: ``` sent opcode=2 (STATUS) -> rcode=0, 701-byte answer, AD=1 sent opcode=15 -> rcode=0, 701-byte answer, AD=1 ``` UPDATE is the dangerous case: a client could believe a record had been changed. RFC 1035 4.1.1 requires NOTIMP, with the opcode **echoed** so the client can tell which outstanding request was rejected. **QCLASS was mixed into the cache key but never validated.** So CHAOS was forwarded upstream as IN: ``` version.bind class=3 (CHAOS) -> rcode=0, 1031 bytes, 6 authority records, AD=1 example.com class=3 -> the validated IN answer, verbatim ``` A recursive server answers only for IN. The lesson generalises: **a field being present in a hash key is not the same as it being checked.** ## Unit tests cannot prove enforcement The first opcode test called `build_response` directly. It passed. Removing the *entire* kernel check that enforces NOTIMP left it **green**, because `build_response` never enforced anything -- it just formats. Mutation testing caught this immediately. Rules that follow: - A test of a *formatter* is not a test of a *decision*. Drive the kernel. - If mutating the code under test does not fail the test, the test is decorative. - **Do not truncate the captured output before deciding.** `cargo test ... | tail -5` cut off the `test result: FAILED` line, so three real mutations read as "not detected" and three load-bearing links looked untested. They were all detected. Capture the whole run, or grep for the marker rather than reading the tail. - Check first whether the enforcement already existed elsewhere. The QR-set check was already in `Query::parse` (`NotAQuery`), so a kernel check for it was dead code; and asserting on the kernel instead of the parser would have passed for the wrong reason. - When a test fails with a confusing number, suspect the test. `flags >> 11` yielded `17` (QR|rcode), not the opcode; the fix was masking `0x7800`. ## Never send more than the client asked for (RFC 6891 6.2.3) There was no truncation at all. A client advertising 512 bytes got 1319 with TC clear, and got the same 1319 whatever it asked for. - Drop records **whole**, in section order, stopping at the first that does not fit. Keeping a later, smaller record in the gap leaves contents that do not match the counts. - Never drop the question: a response whose question does not match is discarded (RFC 1035 4.1.1), turning a working answer into silence. - **Cache the complete answer; fit a copy per client on the way out.** Caching the truncated bytes made every later client receive a partial response. - The **cache-hit path must truncate too.** Returning cached bytes verbatim is the common case and no cold-cache test observes it. ### TC is bit 9 of the flags word -- byte **2** ```rust out[2] |= 0x02; // correct: TC out[3] |= 0x02; // WRONG: sets bit 1 of the low byte, inside the 4-bit RCODE ``` The wrong version turns every NOERROR into SERVFAIL, and only for responses large enough to truncate -- so small fixtures pass and live traffic fails. ### TC cannot be derived from the final size A truncated response *shrinks to fit* and still carries TC. So `len(msg) <= limit` says nothing about whether TC should be set. Two of my own assertions were wrong before the code was. Compare against the **complete** answer's size instead. ## Measuring: fresh process per configuration Reusing one process lets the first request warm the cache, after which every later configuration tests the cache-hit path. Three wiring mutations passed this way. Each size needs its own process, or its own name. To prove a mutation is load-bearing, measure directly rather than trusting a test: ``` correct code : cached@512 -> len=456 tc=1 OK cache-hit bypassed: cached@512 -> len=1319 tc=0 VIOLATION ``` ### Two self-inflicted errors worth remembering - `cargo fmt` reflowed `<< 16` into a multi-line form; a later `str::replace` silently matched nothing, so the change appeared not to take effect. **Assert the replacement applied.** - Moving new functions to sit before `mod tests` deleted the entire test module. When reordering a file, restore from git and diff. ## Header fields that were never validated Four fields the pipeline either ignored or checked only incidentally. All four produced a correct-looking response with **AD=1**, which is the worst shape for a validating resolver. Found by sweeping header fields against the running binary, never by reading code: | Field | Was | Now | |---|---|---| | OPCODE | every opcode resolved as QUERY | NOTIMP, opcode echoed | | QCLASS | mixed into the cache key, never checked | REFUSED unless IN | | EDNS version | read out of the OPT TTL and discarded | BADVERS | | AXFR / IXFR | resolved as lookups -> SERVFAIL | REFUSED | SENTFAIL vs REFUSED matters for transfers: SERVFAIL means "retry later", so a transfer client retries forever against a server that will never comply. ## RFC 6891 OPT TTL layout Counting from the **most significant** bit: ``` |EXTENDED RCODE (8)| VERSION (8) |DO|Z| RCODE (4)| ``` So the extended rcode is bits 31..24 and the version is bits 23..16. Writing BADVERS into bits 16..23 puts it in the VERSION field, so a client decodes your BADVERS as "EDNS version 16". ## Four wrong versions of one test reader Proving the BADVERS fix took four attempts, and every failure looked like "the fix is broken" when the fix was fine. The last reader read ANCOUNT from bytes 4..6 -- which is QDCOUNT -- so it saw ARCOUNT as 0 and returned `None` for every response, including a correct one. - **Dump the bytes.** Do not reason about offsets. `println!("{:02x?}", resp)` plus a Python one-liner resolves in seconds what an hour of reading does not. - Header counts are QD 4..6, AN 6..8, NS 8..10, AR 10..12. Getting this wrong is easy to read as "the three counts after the flags". - A helper returning `Option<u8>` where you expect a value will often be `None` for a reason unrelated to the fix. Check which. ## Windows reserved ports cause 1-in-4 flakes `bind()` on a port in the OS exclusion range returns **PermissionDenied** (10013), not `AddrInUse`: ``` cmd /c "netsh int ipv4 show excludedportrange protocol=tcp" ``` Roughly 1,700 ports are reserved for outbound use, and `bind("127.0.0.1:0")` can hand one back. The symptom is indirect: the resolver logs `TCP listener failed: ... 10013`, correctly keeps serving UDP, and a TCP test then fails with a connection refusal. It reads as resolver flakiness. - Test harnesses must retry on a port that will not bind -- **on any error**, since Linux reports the same collision as `AddrInUse`. - Probe **both** transports before returning a port. A UDP-only probe picks numbers TCP cannot use. - Never let a harness draw a conclusion from a single listener. `transport::run` treats a failed listener as non-fatal by design, which is right in production and wrong to assume in a test. ## Test traps in this codebase