Ironroot resolver
Recursive DNSSEC-validating resolver. Workspace ironroot/: ironroot (userspace),
ironroot-common (no_std shared parser), ironroot-ebpf (XDP L0). Spec IDEA.md.txt.
When to use
Anything touching this resolver: the pipeline, the parsers, the caches, the DNSSEC validator, or the XDP fast path. Read the trap sections before writing a test -- most of the cost here is a wrong assumption about an API, not a hard bug.
Running it
- Bind flag is
--listen <SocketAddr>, NOT--bind. Default127.0.0.1:5353collides with Windows mDNS, so use a high port (e.g.--listen 127.0.0.1:15410). --no-validateexists only with--features diagnostics-no-validate. Without it the field is not compiled in, so a production binary has no bypass to invoke.--trust-anchor <path>points at the RFC 5011 anchor file.
Differential suite (network-dependent):
IRONROOT_UNDER_TEST=127.0.0.1:15411 IRONROOT_REFERENCE=8.8.8.8 \
cargo test -p ironroot --test differential -- --ignored --test-threads=1
Tests and gates
| Suite | Count | Network |
|---|---|---|
| unit | 106 | no |
| conformance | 49 | no |
| properties (proptest) | 16 | no |
| concurrency | 9 | partly |
| hermetic e2e | 4 + 10 ignored | no |
| common | 39 | no |
| differential | 9 | yes |
scripts/verify.sh runs 11 gates, logs each to logs/, prints a summary.
--quick, --only <gate>, --list. Treats a compiler warning as a failure:
#![allow(dead_code)] hid a real bug here.
scripts/diagnose.{sh,py} queries a live resolver and prints a full decode -- flags,
counts, a hand-rolled record walk reporting where a bad message stopped parsing, and a
TCP second opinion.
Remote
Private repo: https://github.com/LoopyLuci/ironroot -- branch main, tag v0.1.0.
Four CI jobs: userspace tests, hermetic e2e, the validation-bypass binary grep, and
the TLC model check. The bypass grep is mutation-tested: it fails against a
diagnostics-no-validate build and passes against a secure one. tla2tools.jar is
fetched by scripts/fetch-tla.sh, not committed.
Measure the running resolver; do not read it
Reading the code found nothing. Timing it found three bugs in one pass:
h0.debian.orgtook 14 s, with a visible 10 s gap between two hops. The per-attempt timeout doubled per retry (RFC 1035 §4.2.2) with no total cap, so one dead server cost 1.5 + 3 + 6 s. Now bounded bymax_total_per_server.--upstream-timeout-msreached only the validator, so it had no effect on the iterator -- the part that waits most. It now lives onUpstreamConfig.- UDP datagrams were dropped by the kernel under burst, because the socket used the OS default buffer. A dropped query is indistinguishable from being down. Buffers now requested at 4 MiB; the kernel grants 8.
Probed and found correct, so left alone: TCP, 14 hostile datagrams, flag hygiene across TC/opcode/AA/CD/RD/Z, and the rate limiter (207/193 at defaults).
Two measurement errors that first looked like resolver bugs
Both are worth remembering because each reads as a server fault:
- Firing 400 datagrams in a tight loop tests the socket buffer, not the limiter. The tokio task never gets scheduled, so the buffer absorbs them and the limiter sees nothing. Pace the burst or yield between sends.
- A UDP socket reused after a timeout reports a spurious ICMP reset on Windows. Zero resets on a fresh socket for the same queries. Use one socket per query when measuring.
#[cfg(unix)] code that only ever ran on Linux
Socket buffer tuning took three attempts that all compiled and passed on Windows
while panicking every runtime worker on Linux, because the block was cfg(unix)
and never executed there. UdpSocket::from_std registers a blocking socket,
which tokio permits only where blocking is legal, and an async fn body never is
-- on any platform. Tuning belongs in main, before the runtime exists, where it
is merely a syscall.
The general rule: anything behind a platform cfg has never been executed unless it
was run on that platform. cargo build on Windows is not a test of it.
- DO bit read from the OPT rclass instead of bit 15 of the OPT TTL (RFC 6891 §6.1.2). The XDP path did this, so the L0 key never matched a DO-bearing client: a silent 100% fast-path miss. Userspace had it right; pin the two together.
- Rate limiter after the cache. The limiter is the pipeline's first stage, but
the L1 lookup ran before the pipeline, so cache hits skipped it and a client
asking for one popular cached name got unlimited service.
Kernelnow holds the limiter and checks it before the cache. cacheableaccepted TTL 0 (RFC 2181 §5.2). Storing it is a correctness bug.- Referrals were cacheable and read as negative answers. NOERROR-with-no-answers
is both a referral's shape and NODATA's. Now
cacheablerequiresancount > 0andis_negativerequires an SOA (RFC 2308 §2.1). - The validator built a fresh transport per request, discarding
--upstream-timeout-ms. Found only after the lib/bin split removed#![allow(dead_code)].
Sweep the corpus; don't assert a hand-written list
ironroot/src/bin/differential-corpus.rs compares 121 questions against a reference
and reports every disagreement. Run it before concluding a resolver behaves.
Five comparison traps, each producing false positives before being fixed:
- Never compare AD with the reference. A validating resolver is required to set AD (RFC 4035 3.2.3) and public resolvers do not echo it. Only "never AD on a failure" is a real invariant.
- Record contents are not comparable. Anycast and round-robin return a different
address subset per query --
reddit.comgave five addresses at the reference, two here, both correct. Compare names and types. - The CNAME chain below the queried name may differ. The reference chased Fastly, we chased Akamai. Compare only records owned by the queried name.
- A cold cache exceeds the per-query timeout. Retry once; report as latency, not correctness.
- A public reference throttles a fast loop. Pace it, and stop after three consecutive reference misses instead of reporting the rest as failures.
The harness must not measure itself
Three separate defects all had the same shape: the gate or sweep reported its own limits as resolver failures, and each read convincingly like a resolver bug.
- The corpus sweep tripped the resolver's own rate limiter. 121 questions in quick succession from one client address exceed the default 100 qps / 200 burst, and the rejections were counted as timeouts: 66/121 agreed, with 54 "no bytes back". The resolver was throttling the test, not failing. Raise the limits for the harness resolver only; production defaults are correct and stay tested.
- The sweep's 8s per-query timeout was measuring its own pacing, not resolution time. Four names timed out that resolve in 0.1-1.3s standalone. At 20s: 121/121.
cargo cleannever ran because a barecargodoes not exist on this host, onlycargo.exe, so the gate meant to prevent inspecting a stale binary inspected a stale binary.
Before believing a "resolver failure", ask what the harness was doing when it happened. A timeout and a rejection are both harness-shaped.
A Windows socket error is not a fatal one
recv_from(&mut buf).await? in the UDP accept loop removed the listener on the first
recoverable error. Windows reports WSAECONNRESET (10054) on recv_from after an ICMP
port-unreachable from an earlier send -- routine, and caused by any client that
vanishes between our reply and its retransmit.
The symptom is deceptive: the process is alive and still holds the socket, so health checks pass, but nothing is answered any more. It showed up as the corpus sweep agreeing on 80 of 121 questions and then reporting "no bytes back" for every remainder -- which reads as rate limiting, not a dead listener.
Count transient errors and continue; log the first and every hundredth so a persistent
fault stays visible. Give up on a bound number of consecutive errors rather than
spinning forever. ConnectionReset, Interrupted, WouldBlock and TimedOut are all
recoverable; anything else is a genuine fault.
The BPF gate was judging a script that never ran
wsl.exe inherits Git Bash's MSYS path translation, which rewrites a leading /mnt/z/...
argument into C:/Program Files/Git/mnt/z/.... Bash inside WSL was handed a path that
does not exist, the gate judged the resulting log, and it reported "object identity
unproven" -- a verdict about a script that never executed. That is why the same failure
survived every loader fix: the loader was never being tested.
MSYS_NO_PATHCONV=1 wsl.exe -d Debian --exec bash /mnt/z/.../verify-bpf.sh
Per-command, not exported -- other commands in the script rely on the conversion.
A gate that cannot run must fail loudly, and this one failed quietly. "Object identity unproven" reads like a stale-object problem; the real cause was that the verifier never started. Check what the gate is executing before believing what it reports about it.
Two more defects hid behind that one:
awk'sstrtonumis a gawk extension; Debian ships mawk, so the instruction count printed empty and the run looked like an identity failure again. Preferprintf '%d'for hex, and degrade to a visible "unknown" rather than to nothing.bpfel-unknown-noneis a low-tier target with no prebuilt artifacts, sorustup target addrefuses it. It builds only with-Z build-std=core(andrust-src). The gate already passed that flag -- it simply had not run.
What the verifier actually says, in the order the errors appeared
processed 0 insns on a 2896-instruction program is the signature of
check_subprogs() failing: it walks every BPF-to-BPF call target and requires each
subprogram's last instruction to be an exit or unconditional jump, and it runs
before any instruction is verified. So the error names "last insn" while the main
program's last instruction is correctly an exit.
Three loader defects sat behind that, each invisible until the previous was fixed:
A call target ignored the symbol's offset within its section.
.textholds memcpy at 0, memmove at 0x58 and memset at 0x120; copying the whole section and aiming every call at its start sendsmemsettomemcpy. The call reaches a real subprogram, so the program loads far enough to walk the CFG and reports the first instruction no path reaches -- memmove's, since only memset was called.unreachable insn 2907named a function nobody was looking at. Copy fromsh_offset + st_value, notsh_offset..text.unlikely.holds two adjacent single-instruction panic subprograms. The first becomes an exit under the panic rewrite, so the CFG walk terminates there and the second is never entered;check_cfgdemands every instruction be explored. Drop the second -- but scope that to the panic block. A general "drop the last instruction" rule truncatesmemset, an eight-instruction function whose loop then cannot be explored. That mistake cost a round.The trailing
exitappended to fix error 1 became the cause of error 2. Once every appended subprogram terminates in its own right, nothing falls through to a trailing instruction. The requirement and the old fix were one idea seen from two sides: a program whose last instruction is both reachable and terminal.last insn is not an exit or jmp -> a subprogram, not the main program unreachable insn 2907 -> memset aimed at memcpy unreachable insn 2905 -> the trailing exit processed 2445 insns -> the verifier is finally checking
A panic subprogram that calls itself has no terminating instruction at all, so
appending an exit past the end of the program can never satisfy the check. Patching only
the goto -1 half of a call 0x0 + goto -0x1 pair changes nothing.
verify-bpf.sh rebuilds the loader from the committed vm/load.c on every run, so a
binary staged into the VM is overwritten. Know that before concluding a change had no
effect -- one experiment was void for exactly this reason.
L0 is cut: not shipped, gate skips with a reason
L0 was cut from shipping scope on 2026-10-04 (068fcf4). The bpf gate now
skips with the reason rather than failing, prints the verifier's own words so the
evidence stays live, and fails hard under IRONROOT_BPF_STRICT=1. Final state:
11 PASS, 1 SKIP, "all gates passed".
docs/bpf-loader-notes.md is the handoff: the three loader bugs, the
instruction-count progression, the failed experiments, and the stop point. The BPF crate,
vm/ and the loader stay in the tree, and CI has an l0-builds job so "it still builds"
stays true.
Reasoning worth not relitigating: L0 is an optimisation, not a security control. The userspace path validates signed, unsigned-by-proof, forged and downgrade cases end to end. A permanently red gate with an unactionable verdict is worse than a documented one -- that is exactly how the MSYS path-mangling bug hid for days behind "object identity unproven".
Harness traps that cost real time
bind("127.0.0.1:0")can return a port inside a Windows excluded range. The TCP bind then fails withos error 10013("access permissions"), not "in use", and retrying cannot escape it because the OS keeps offering the same reserved range. Draw test ports from a fixed base instead. Mutation-verified: 22 of 30 runtime tests fail with the OS-assigned port.- A section symbol has an empty name and
st_value0, so a relocation naming.textcarries no addend. That makesbounded<1>andbounded<2>indistinguishable to the loader. Caching "append each section once" is correct in principle and unfixable with section-symbol relocations -- it aims every call at offset 0 and leaves the other monomorphisation unreachable. Do not go down this path without per-symbol relocations. vm/verify-bpf.shrebuilds the loader from the committedvm/load.cevery run, so a binary staged into the VM by hand is silently overwritten.- Do not delete an instruction from the middle of a BPF program. Every following relative
jump breaks (
jump out of range); you must fix up all later offsets. - The harness's own identity check can measure a stale file. It extracted the initramfs from
a hardcoded path while the image is written to a run-specific directory. Cross-check
PROG_MD5(the object the verifier actually loaded) against the host hash before believing any "identity unproven" verdict. - Cosmetic edits can change BPF codegen. Removing an
#[inline(always)]that sat inside a doc-comment block changed.textfrom 3 functions to 7 (extrabounded<N>monomorphisations) and regressed 2445 instructions toprocessed 0 insns. Verify a BPF build after any change to that file, however unrelated it looks.
Verify claims against source before writing them down
A plausible number entered a status report, got copied forward, and became a "fact" that was
never in the code: a 6,000 ms total-per-server budget that did not exist. Only
UPSTREAM_TIMEOUT (1,500 ms per attempt) was real, with retries counted by
MAX_REFERRALS. It was about to be documented when a check against the source caught it.
Same shape as the codegen trap, in prose instead of instructions: something asserted early, copied forward, and never re-derived. Check behavioural claims against the source before documenting them, not against memory or an earlier report.
Mutation tests: check the mutation removed the property
Replacing match timed.await { Ok(o) => o, Err(_) => ... } with unwrap_or_else left the
timeout fully intact, and the test passed. Replacing the whole call with a bare .await
was the mutation that actually removed the behaviour -- and the test then failed, reporting
the budget did not stop the query: 30.008s.
A mutation that leaves the property intact proves nothing. Ask what the mutant would have to look like for the property to be gone, and check the test fails on that.
Budgets: an attempt timeout is not a query timeout
upstream_timeout bounds one attempt; retries are counted, so the query ceiling was
attempts x timeout. total_budget (3,000 ms, --total-budget-ms) is the missing limit:
the pipeline runs inside tokio::time::timeout, and expiry gives SERVFAIL with AD clear and
nothing cached. Cancelling the future drops the in-flight recv_from rather than leaking a
task or socket.
Implementation note: the stages had to move into run_pipeline because an async fn body
cannot be borrowed across a timeout -- the whole loop has to live in its own future.
Iteration and validation do reach the right servers
Checked rather than assumed, since a budget over a broken resolver treats a symptom:
- The iterator genuinely iterates: root -> TLD -> authoritative, 70 referral references, and a live 121/121 corpus against the real root servers.
- The validator has zero
ROOT_SERVERSreferences. It uses the configuredUpstreamConfigand a shared delegation cache (33 references), so DS and DNSKEY go to the right server.
Set resource ceilings from the maximum, never the median
The total_budget default was a judgement call; measuring it showed the guess was wrong in the
dangerous direction. differential-corpus prints the cold-path distribution (first-try answers
only -- a retry's cost is harness load, not resolver latency):
budget p50 p90 p99 max SERVFAILs
3,000ms 399ms 983ms 2s 3.005s 3
30,000ms 398ms 914ms 1s 2.087s 0
At 3 s the max lands at 3.005 s -- the ceiling firing -- with SERVFAILs. At 30 s the same sweep has none and tops out at 2.087 s, the true healthy maximum. The default is now 5 s (~2.4x margin), mutation-verified by reverting to 3 s.
The reasoning that produced the wrong answer is worth keeping as an anti-pattern: "typical latency is well under 100 ms, so 1.5 s is generous." A p50 says nothing about the maximum. A resource ceiling set below the slowest healthy resolution converts a slow-but-working server into a failure for every client behind it.
Raising a ceiling is safer than lowering one. When tuning, err upward, and re-measure.
This generalises past timeouts: any resource limit set below the observed healthy maximum silently converts working systems into failures, and it fails cleanly -- a ceiling that fires correctly looks identical to a ceiling set wrongly, in every aggregate. Guard the default with a test asserting both bounds; the upper bound catches an unbounded limit, the lower catches one that fires on healthy work. Assert against the measurement in the message, so the next tuner sees where the number came from.
Async ceilings do not cover CPU work
tokio::time::timeout fires only at an .await. A module doing sustained compute is not
interrupted mid-operation. Fine for this resolver today (DNSSEC verification is microseconds to
low milliseconds; stages are I/O-bound on DS/DNSKEY fetches), but a future compute-bound stage
needs its own bounds.
Reading a BPF refusal
processed 0 insns is the signature of check_subprogs rejecting a subprogram before
instruction verification begins -- the message says "last insn" while the main program's last
instruction is correctly an exit. A non-zero count means the verifier is genuinely checking.
Lesson repeated twice: the mechanism in a bug report may be right while the named culprit
is wrong. Verify the named thing exists. One report blamed aya-log-ebpf's info! --
the source contained no info! and no aya_log_ebpf reference at all. Reading the
disassembly of the actual object settled it in one step.
Two load.c facts worth keeping: a call immediate is a signed instruction distance
from the calling instruction, and code from other sections must be copied into the
program array with insn_cnt covering the appended code.
verify-bpf.sh rebuilds the BPF object from the Windows tree every run. The
loader that actually executes it lives inside the initramfs at
/home/ironroot/vm/root_base/loader and is not rebuilt. Days of edits to
vm/load.c were silently untested, and the gate kept reporting the same error the
whole time. After touching load.c:
cp vm/load.c ~/vm/ && cd ~/vm && gcc -O2 -static -o load load.c && cp load root_base/loader
Three bugs lived in that untested file. A call immediate is a signed instruction
distance from the calling instruction, not a byte offset and not a file offset — the
old code wrote symbol value + section file offset, which is short by the program's own
start offset and puts .text.unlikely. outside the program entirely
("call to invalid destination"). Code in another section must be copied into the
program array, since a BPF program is one contiguous block. And insn_cnt has to count
the appended code, or the destinations exist outside the submitted range and produce the
same error — which is what made the second bug hard to spot.
Always print the value actually submitted, not the variable that happens to be nearby. A diagnostic reading a stale variable is worse than none: it reports a plausible number and hides the effect of the change.
Never test against a stale binary
A temporary debug line failed to compile, so the "fix worked" run actually exercised
the previous artifact and reported a pass. cargo build returning non-zero is not an
optional detail: check the exit status before believing any result from a rebuilt
binary, and prefer a mutation that must fail to a test that must pass -- a green test
that never ran proves nothing.
The same trap when a mutation silently does not apply: assert the mutated text differs from the original before running anything, or the "detected" result is fiction.
Gate wiring: a gate that cannot run must say so
The corpus sweep was added to the GATES array but never dispatched. It printed
unknown gate: corpus and the run still reported "all gates passed". A gate that
cannot run must fail, not skip quietly -- a silent skip is a green tick it did not earn.
Related, all found the hard way: A=x B=y on one line is a single command taking A as
its name, so neither variable is set. --bin foo builds only foo, leaving
sibling binaries absent so later checks run against a missing file -- and a missing
executable reads as a passing security check. [ -f "X:/..." ] is always false from a
POSIX shell, so Windows-form paths silently miss files that exist; and guessing the
POSIX form (/X/... rather than MSYS's /mnt/x/...) is worse than not converting,
because an unverified guess looks like a real path.
DNSSEC verdicts: absence of evidence is not forgery
The two worst bugs were verdict-mapping errors, in different code paths:
| Situation | Correct | Was |
|---|---|---|
| Unsigned zone, answer has no RRSIG | Insecure | Err -> SERVFAIL |
| Unsigned zone, NODATA with no SOA | Insecure | Ok(Some(Bogus)) -> SERVFAIL |
Missing SOA record for NODATA is only a violation in a signed zone. In an
unsigned zone the server is not obliged to send one. github.com publishes no DS, so
both its plain answers and its NODATA answers were refused -- one of the most-visited
names on the Internet.
The guard stays narrow: zero RRSIG records anywhere means Insecure. Anything carrying signatures that fails to parse still fails closed. Fix one arm and check the other -- the first fix did not cover the second.
Mutation-test every verdict fix. Reverting the Bogus-path fix left the suite green until unsigned-NODATA coverage existed.
Distinguish a broken zone from a resolver bug
A sweep flagged go.dns.pl as SERVFAIL. Chasing it: dns.pl publishes a DS but
serves no DNSKEY -- an unsigned island -- and Google and Cloudflare SERVFAIL on it
too. Ironroot answers dns.pl with AD=1, more strictly than either. Broken zone,
not a bug; SERVFAIL was right. Check whether independent references fail the same way
before "fixing" anything.
Two gaps found by probing the header, not the code
Reading the resolver found nothing in these two places; a sweep of header fields did. Both produced a correct-looking response with the AD bit set, which is the worst kind of bug for a validating resolver: the answer looked verified.
No opcode validation existed at all. Every opcode was resolved as QUERY:
sent opcode=2 (STATUS) -> rcode=0, 701-byte answer, AD=1
sent opcode=15 -> rcode=0, 701-byte answer, AD=1
UPDATE is the dangerous case: a client could believe a record had been changed. RFC 1035 4.1.1 requires NOTIMP, with the opcode echoed so the client can tell which outstanding request was rejected.
QCLASS was mixed into the cache key but never validated. So CHAOS was forwarded upstream as IN:
version.bind class=3 (CHAOS) -> rcode=0, 1031 bytes, 6 authority records, AD=1
example.com class=3 -> the validated IN answer, verbatim
A recursive server answers only for IN. The lesson generalises: a field being present in a hash key is not the same as it being checked.
Unit tests cannot prove enforcement
The first opcode test called build_response directly. It passed. Removing the
entire kernel check that enforces NOTIMP left it green, because
build_response never enforced anything -- it just formats. Mutation testing
caught this immediately.
Rules that follow:
- A test of a formatter is not a test of a decision. Drive the kernel.
- If mutating the code under test does not fail the test, the test is decorative.
- Do not truncate the captured output before deciding.
cargo test ... | tail -5cut off thetest result: FAILEDline, so three real mutations read as "not detected" and three load-bearing links looked untested. They were all detected. Capture the whole run, or grep for the marker rather than reading the tail. - Check first whether the enforcement already existed elsewhere. The QR-set check
was already in
Query::parse(NotAQuery), so a kernel check for it was dead code; and asserting on the kernel instead of the parser would have passed for the wrong reason. - When a test fails with a confusing number, suspect the test.
flags >> 11yielded17(QR|rcode), not the opcode; the fix was masking0x7800.
Never send more than the client asked for (RFC 6891 6.2.3)
There was no truncation at all. A client advertising 512 bytes got 1319 with TC clear, and got the same 1319 whatever it asked for.
- Drop records whole, in section order, stopping at the first that does not fit. Keeping a later, smaller record in the gap leaves contents that do not match the counts.
- Never drop the question: a response whose question does not match is discarded (RFC 1035 4.1.1), turning a working answer into silence.
- Cache the complete answer; fit a copy per client on the way out. Caching the truncated bytes made every later client receive a partial response.
- The cache-hit path must truncate too. Returning cached bytes verbatim is the common case and no cold-cache test observes it.
TC is bit 9 of the flags word -- byte 2
out[2] |= 0x02; // correct: TC
out[3] |= 0x02; // WRONG: sets bit 1 of the low byte, inside the 4-bit RCODE
The wrong version turns every NOERROR into SERVFAIL, and only for responses large enough to truncate -- so small fixtures pass and live traffic fails.
TC cannot be derived from the final size
A truncated response shrinks to fit and still carries TC. So len(msg) <= limit
says nothing about whether TC should be set. Two of my own assertions were wrong
before the code was. Compare against the complete answer's size instead.
Measuring: fresh process per configuration
Reusing one process lets the first request warm the cache, after which every later configuration tests the cache-hit path. Three wiring mutations passed this way. Each size needs its own process, or its own name.
To prove a mutation is load-bearing, measure directly rather than trusting a test:
correct code : cached@512 -> len=456 tc=1 OK
cache-hit bypassed: cached@512 -> len=1319 tc=0 VIOLATION
Two self-inflicted errors worth remembering
cargo fmtreflowed<< 16into a multi-line form; a laterstr::replacesilently matched nothing, so the change appeared not to take effect. Assert the replacement applied.- Moving new functions to sit before
mod testsdeleted the entire test module. When reordering a file, restore from git and diff.
Header fields that were never validated
Four fields the pipeline either ignored or checked only incidentally. All four produced a correct-looking response with AD=1, which is the worst shape for a validating resolver. Found by sweeping header fields against the running binary, never by reading code:
| Field | Was | Now |
|---|---|---|
| OPCODE | every opcode resolved as QUERY | NOTIMP, opcode echoed |
| QCLASS | mixed into the cache key, never checked | REFUSED unless IN |
| EDNS version | read out of the OPT TTL and discarded | BADVERS |
| AXFR / IXFR | resolved as lookups -> SERVFAIL | REFUSED |
SENTFAIL vs REFUSED matters for transfers: SERVFAIL means "retry later", so a transfer client retries forever against a server that will never comply.
RFC 6891 OPT TTL layout
Counting from the most significant bit:
|EXTENDED RCODE (8)| VERSION (8) |DO|Z| RCODE (4)|
So the extended rcode is bits 31..24 and the version is bits 23..16. Writing BADVERS into bits 16..23 puts it in the VERSION field, so a client decodes your BADVERS as "EDNS version 16".
Four wrong versions of one test reader
Proving the BADVERS fix took four attempts, and every failure looked like "the fix
is broken" when the fix was fine. The last reader read ANCOUNT from bytes 4..6 --
which is QDCOUNT -- so it saw ARCOUNT as 0 and returned None for every response,
including a correct one.
- Dump the bytes. Do not reason about offsets.
println!("{:02x?}", resp)plus a Python one-liner resolves in seconds what an hour of reading does not. - Header counts are QD 4..6, AN 6..8, NS 8..10, AR 10..12. Getting this wrong is easy to read as "the three counts after the flags".
- A helper returning
Option<u8>where you expect a value will often beNonefor a reason unrelated to the fix. Check which.
Windows reserved ports cause 1-in-4 flakes
bind() on a port in the OS exclusion range returns PermissionDenied (10013),
not AddrInUse:
cmd /c "netsh int ipv4 show excludedportrange protocol=tcp"
Roughly 1,700 ports are reserved for outbound use, and bind("127.0.0.1:0") can
hand one back. The symptom is indirect: the resolver logs
TCP listener failed: ... 10013, correctly keeps serving UDP, and a TCP test then
fails with a connection refusal. It reads as resolver flakiness.
- Test harnesses must retry on a port that will not bind -- on any error, since
Linux reports the same collision as
AddrInUse. - Probe both transports before returning a port. A UDP-only probe picks numbers TCP cannot use.
- Never let a harness draw a conclusion from a single listener.
transport::runtreats a failed listener as non-fatal by design, which is right in production and wrong to assume in a test.
Test traps in this codebase
ttl::for_each_section(msg, start, count, f):startis how many records to skip, not a section ordinal. Use(msg, 0, ancount)for answers,(msg, ancount, nscount)for authority. Passing1silently reads the wrong records with no error.wire::resolve_namereturns labels without the root byte; decode withdnsname::wire_to_name_labels.Question.qnamedoes include it, soName::from_wireis right there.- In an integration test
crate::is the test binary, not the library. Fixtures must sayironroot::wire::.... Kernelis single-owner;Servicesis notClone. UseKernel::shared_services()to keep observing services aftertransport::runconsumes the kernel.Kernelitself is nowClone(allArcfields).Kernel::handleis async: each test thread needs its own current-thread runtime.clapreports an unknown flag on stderr; a shell gate checkingbinary --flag 2>&1 | grepmust capture both or it fails a correct binary.- In a
{ }group,exitends the whole script. Use( ... )for a subshell. - Generate proptest inputs from
prop::sample::select(ALPHABET), notany::<char>()plus a filter: filtering to ASCII rejects ~95% of draws and proptest aborts with "Too many local rejects", which looks like a pass and is not. MAX_TTLclamps a 136-year TTL to one day deliberately. A property assertingcached_ttl >= record_ttlfails above 86400 and looks like a resolver bug.- Run new suites three times. A stress test whose final assertion does a real upstream lookup will flake, because the test just exhausted that path.
- A readiness poll must use a short timeout.
TestServer::wait_until_readyonce used the same 10s read timeout as a real query inside a 10s deadline, so one unanswered probe ate the whole budget and the loop could not retry. It passed on Windows and failed on the CI runner only because startup was slower (21.5s of doomed waiting versus 0.5s). Hencetry_query_within(msg, timeout): 200ms to poll, 10s to ask a real question. - Green locally is not green on Linux. CI runs ubuntu; the BPF toolchain lives
in Debian WSL. Check both before pushing, and install
rustfmt/clippyfor the stable toolchain in WSL, not just nightly, orcargo fmtfails there. cargo clippyfinds real defects, not just style: it caught a retry loop that returned on its first iteration and so could never retry, and au16 | u8mismatch when a "redundant" cast was removed. Trust its-- -D warningsrun.- The BPF target's nightly rejects
fn f<N: usize>; writefn f<const N: usize>.
BPF verifier
Use vm/bisect_bpf.py, vm/bisect_reads.py, vm/bisect_bounds.py -- each edits,
rebuilds in WSL, reports, reverts. A round costs ~4 min and eliminates a category.
Rules, all hard-won:
- Every packet access must be a constant offset from one base pointer.
- Integer bounds checks (
offset < len) prove nothing. Compare a derived pointer againstpkt_end. Arithmetic onpkt_endis forbidden; derive one pointer and add a constant to it. - Any
[u8; N]with a fill byte is a memset, pulling compiler-builtins into.text, which aya concatenates into the program. UseMaybeUninit. - Prefer a const ladder (64/32/16/8/4/2/1): a 13-byte name is shorter than a 64-byte window and a 255-byte name overflows the 512-byte stack.
read_u8,read_be16,write_be16must all go throughread_span/write_span. A baregetor two separate byte reads fails verification while passing every host test.
Harness: ~/vm in Debian WSL, vm/verify-bpf.sh MD5-checks the object against the
copy embedded in the initramfs. Never trust a verdict without that match.
Status: not accepted, but the cause is now pinned. Staging the QNAME into a stack buffer does clear the verifier's packet-access complaint. The blocker moved to a different wall, and it is a specific one:
The write into the staging buffer must leave no panic call. aya concatenates
.text.unlikelyinto the program, so any panic the compiler cannot prove unreachable becomes a trailingcallthe loader cannot resolve -- and the symptom is "last insn is not an exit or jmp", which says nothing about panics.
Five shapes tried, one failure each:
| shape | leaves |
|---|---|
copy_from_slice at a runtime length |
memmove |
buf[i + k] |
a bounds-check panic |
buf.get_mut(a..b) |
a range-check panic |
buf.as_mut_ptr().add(i) cast to [u8; N] |
an alignment panic |
nested [[u8; BLOCK]; N] |
two index checks |
The nested array ships (correct, tested, two panics left). Next: a fixed-width
copy_from_slice::<$n> between two arrays of the same compile-time width -- no
runtime length anywhere in it, so the remaining checks are over constants.
Also learned the hard way:
- The ladder's width must come from what the packet offers, not the buffer's remaining space. Backwards stages nothing for any query shorter than STAGE.
- A name longer than one block crosses the boundary; dropping the spilled bytes silently changes the hashed name, so the key stops matching the userspace formula. Mutation testing caught this one.
- STAGE must fit the 512-byte frame alongside
build_reply's buffers: 64 works, 128 does not. A longer name than STAGE is an L0 miss, which is correct. - The staged length is a lower bound when the packet is short; the walk must stop there rather than read uninitialised stack.
rmthe panic and the check becomes redundant: the compression-pointer guard is subsumed bylabel_len > 63. Kept and documented as defence in depth with the mutation result, rather than implied to be load-bearing.
vm/bisect_*.py are the tools that got here; a round is ~4 min.
Hermetic end-to-end: per-resolver config, and what it unblocked
A mock root and a mock authority must be different sockets -- the resolver expects a referral from one and a final answer from the other.
UpstreamConfig carries the root set and the delegation port on the resolver
itself, reached via RuntimeConfig and Services. It must be per-resolver, not
process-global: env vars were tried twice and both failed -- read once per process
pinned every test to the first test's root (a silent false pass), read live each
test overwrote the previous mock (wholesale timeout).
The root's port and the delegation port are different fields. A glue record carries only an IP, so the delegation port is ours to choose; the root's port is part of a real address we were handed. Collapsing them points root queries at the authority's socket.
Defects that surfaced only once the suite could run:
TestServer::_mockwasOption, so a secondwith_mocksilently dropped the first -- the mock authority, whose socket closed on drop. Indistinguishable from a resolver that cannot reach its delegated server. It is aVecnow.- A negative answer was treated as a referral. Empty answer + populated authority is both shapes; the code discriminated on section counts. NXDOMAIN carries an SOA there (RFC 2308 2.1), so it went back to the root and died as a referral loop. Check for actual NS records.
resolve_namereturns labels, not encoded bytes. Reading qtype at12 + resolve_name(..).len()silently decodes 0.- A mock that echoes a hardcoded qtype gets its referral rejected as a mismatch -- correctly. Echo the query's type.
Still ignored, each with its reason: two need an NSEC proof of DS absence to establish an insecure delegation (needs a signed parent zone); one needs per-query rcodes from the mock. The resolver is right in the first two -- unable to distinguish unsigned from forged, it says bogus, which is correct.
A mock can assert a secure verdict. It was previously assumed it could not, which is why the suite could only ever pin "unverifiable data is withheld" and never "signed and valid" -- the distinction those tests exist to draw. See below.
Mocks must serve TCP, and truncation must be transport-specific
RFC 1035 §4.2.1 makes the resolver retry a truncated UDP response over TCP, so a UDP-only mock cannot test it at all. Serve UDP and TC
…(truncated)