# Ironroot Resolver

> Use for the Ironroot DNSSEC resolver in DNS-Resolver.

- Skill: `loopyluci/ironroot-resolver` (Agent Skill)
- Install (CLI): `npx get-skill add loopyluci/ironroot-resolver`
- Raw SKILL.md: https://api.skillmd.com/api/skills/loopyluci/ironroot-resolver/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: LoopyLuci (https://skillmd.com/u/loopyluci)
- Updated: 2026-11-15
- Page: https://skillmd.com/skills/loopyluci/ironroot-resolver

---


# Ironroot resolver

Recursive DNSSEC-validating resolver. Workspace `ironroot/`: `ironroot` (userspace),
`ironroot-common` (no_std shared parser), `ironroot-ebpf` (XDP L0). Spec `IDEA.md.txt`.

## When to use

Anything touching this resolver: the pipeline, the parsers, the caches, the DNSSEC
validator, or the XDP fast path. Read the trap sections before writing a test --
most of the cost here is a wrong assumption about an API, not a hard bug.

## Running it

- Bind flag is `--listen <SocketAddr>`, NOT `--bind`. Default `127.0.0.1:5353` collides
  with Windows mDNS, so use a high port (e.g. `--listen 127.0.0.1:15410`).
- `--no-validate` exists only with `--features diagnostics-no-validate`. Without it the
  field is not compiled in, so a production binary has no bypass to invoke.
- `--trust-anchor <path>` points at the RFC 5011 anchor file.

Differential suite (network-dependent):

    IRONROOT_UNDER_TEST=127.0.0.1:15411 IRONROOT_REFERENCE=8.8.8.8 \
      cargo test -p ironroot --test differential -- --ignored --test-threads=1

## Tests and gates

| Suite | Count | Network |
|---|---|---|
| unit | 106 | no |
| conformance | 49 | no |
| properties (proptest) | 16 | no |
| concurrency | 9 | partly |
| hermetic e2e | 4 + 10 ignored | no |
| common | 39 | no |
| differential | 9 | yes |

`scripts/verify.sh` runs 11 gates, logs each to `logs/`, prints a summary.
`--quick`, `--only <gate>`, `--list`. Treats a compiler *warning* as a failure:
`#![allow(dead_code)]` hid a real bug here.

`scripts/diagnose.{sh,py}` queries a live resolver and prints a full decode -- flags,
counts, a hand-rolled record walk reporting where a bad message stopped parsing, and a
TCP second opinion.

## Remote

Private repo: https://github.com/LoopyLuci/ironroot -- branch `main`, tag `v0.1.0`.
Four CI jobs: userspace tests, hermetic e2e, the validation-bypass binary grep, and
the TLC model check. The bypass grep is mutation-tested: it fails against a
`diagnostics-no-validate` build and passes against a secure one. `tla2tools.jar` is
fetched by `scripts/fetch-tla.sh`, not committed.

## Measure the running resolver; do not read it

Reading the code found nothing. Timing it found three bugs in one pass:

- `h0.debian.org` took **14 s**, with a visible 10 s gap between two hops. The
  per-attempt timeout doubled per retry (RFC 1035 §4.2.2) with no total cap, so
  one dead server cost 1.5 + 3 + 6 s. Now bounded by `max_total_per_server`.
- `--upstream-timeout-ms` reached only the validator, so it had no effect on the
  iterator -- the part that waits most. It now lives on `UpstreamConfig`.
- UDP datagrams were **dropped by the kernel** under burst, because the socket
  used the OS default buffer. A dropped query is indistinguishable from being
  down. Buffers now requested at 4 MiB; the kernel grants 8.

Probed and found correct, so left alone: TCP, 14 hostile datagrams, flag hygiene
across TC/opcode/AA/CD/RD/Z, and the rate limiter (207/193 at defaults).

### Two measurement errors that first looked like resolver bugs

Both are worth remembering because each reads as a server fault:

- **Firing 400 datagrams in a tight loop tests the socket buffer, not the
  limiter.** The tokio task never gets scheduled, so the buffer absorbs them and
  the limiter sees nothing. Pace the burst or yield between sends.
- **A UDP socket reused after a timeout reports a spurious ICMP reset on
  Windows.** Zero resets on a fresh socket for the same queries. Use one socket
  per query when measuring.

### `#[cfg(unix)]` code that only ever ran on Linux

Socket buffer tuning took three attempts that all compiled and passed on Windows
while panicking every runtime worker on Linux, because the block was `cfg(unix)`
and never executed there. `UdpSocket::from_std` registers a *blocking* socket,
which tokio permits only where blocking is legal, and an `async fn` body never is
-- on any platform. Tuning belongs in `main`, before the runtime exists, where it
is merely a syscall.

The general rule: anything behind a platform cfg has never been executed unless it
was run on that platform. `cargo build` on Windows is not a test of it.

- **DO bit read from the OPT rclass** instead of bit 15 of the OPT TTL (RFC 6891
  §6.1.2). The XDP path did this, so the L0 key never matched a DO-bearing client:
  a silent 100% fast-path miss. Userspace had it right; pin the two together.
- **Rate limiter after the cache.** The limiter is the pipeline's first stage, but
  the L1 lookup ran *before* the pipeline, so cache hits skipped it and a client
  asking for one popular cached name got unlimited service. `Kernel` now holds the
  limiter and checks it before the cache.
- **`cacheable` accepted TTL 0** (RFC 2181 §5.2). Storing it is a correctness bug.
- **Referrals were cacheable and read as negative answers.** NOERROR-with-no-answers
  is both a referral's shape and NODATA's. Now `cacheable` requires `ancount > 0`
  and `is_negative` requires an SOA (RFC 2308 §2.1).
- **The validator built a fresh transport per request**, discarding
  `--upstream-timeout-ms`. Found only after the lib/bin split removed
  `#![allow(dead_code)]`.

## Sweep the corpus; don't assert a hand-written list

`ironroot/src/bin/differential-corpus.rs` compares 121 questions against a reference
and reports **every** disagreement. Run it before concluding a resolver behaves.

Five comparison traps, each producing false positives before being fixed:

- **Never compare AD with the reference.** A validating resolver is *required* to set
  AD (RFC 4035 3.2.3) and public resolvers do not echo it. Only "never AD on a
  failure" is a real invariant.
- **Record contents are not comparable.** Anycast and round-robin return a different
  address subset per query -- `reddit.com` gave five addresses at the reference, two
  here, both correct. Compare **names and types**.
- **The CNAME chain below the queried name may differ.** The reference chased Fastly,
  we chased Akamai. Compare only records owned by the queried name.
- **A cold cache exceeds the per-query timeout.** Retry once; report as latency, not
  correctness.
- **A public reference throttles a fast loop.** Pace it, and stop after three
  consecutive reference misses instead of reporting the rest as failures.

## The harness must not measure itself

Three separate defects all had the same shape: the gate or sweep reported its own
limits as resolver failures, and each read convincingly like a resolver bug.

- **The corpus sweep tripped the resolver's own rate limiter.** 121 questions in quick
  succession from one client address exceed the default 100 qps / 200 burst, and the
  rejections were counted as timeouts: 66/121 agreed, with 54 "no bytes back". The
  resolver was throttling the test, not failing. Raise the limits for the harness
  resolver only; production defaults are correct and stay tested.
- **The sweep's 8s per-query timeout was measuring its own pacing**, not resolution
  time. Four names timed out that resolve in 0.1-1.3s standalone. At 20s: 121/121.
- **`cargo clean` never ran** because a bare `cargo` does not exist on this host, only
  `cargo.exe`, so the gate meant to prevent inspecting a stale binary inspected a stale
  binary.

Before believing a "resolver failure", ask what the harness was doing when it happened.
A timeout and a rejection are both harness-shaped.

## A Windows socket error is not a fatal one

`recv_from(&mut buf).await?` in the UDP accept loop removed the listener on the first
recoverable error. Windows reports `WSAECONNRESET` (10054) on `recv_from` after an ICMP
port-unreachable from an earlier send -- routine, and caused by any client that
vanishes between our reply and its retransmit.

The symptom is deceptive: the process is alive and still holds the socket, so health
checks pass, but nothing is answered any more. It showed up as the corpus sweep
agreeing on 80 of 121 questions and then reporting "no bytes back" for every
remainder -- which reads as rate limiting, not a dead listener.

Count transient errors and continue; log the first and every hundredth so a persistent
fault stays visible. Give up on a bound number of consecutive errors rather than
spinning forever. `ConnectionReset`, `Interrupted`, `WouldBlock` and `TimedOut` are all
recoverable; anything else is a genuine fault.

## The BPF gate was judging a script that never ran

`wsl.exe` inherits Git Bash's MSYS path translation, which rewrites a leading `/mnt/z/...`
argument into `C:/Program Files/Git/mnt/z/...`. Bash inside WSL was handed a path that
does not exist, the gate judged the resulting log, and it reported **"object identity
unproven"** -- a verdict about a script that never executed. That is why the same failure
survived every loader fix: the loader was never being tested.

```
MSYS_NO_PATHCONV=1 wsl.exe -d Debian --exec bash /mnt/z/.../verify-bpf.sh
```

Per-command, not exported -- other commands in the script rely on the conversion.

**A gate that cannot run must fail loudly, and this one failed quietly.** "Object identity
unproven" reads like a stale-object problem; the real cause was that the verifier never
started. Check what the gate is *executing* before believing what it reports about it.

Two more defects hid behind that one:

- `awk`'s `strtonum` is a **gawk extension**; Debian ships mawk, so the instruction count
  printed empty and the run looked like an identity failure again. Prefer `printf '%d'`
  for hex, and degrade to a visible "unknown" rather than to nothing.
- `bpfel-unknown-none` is a **low-tier target with no prebuilt artifacts**, so `rustup
  target add` refuses it. It builds only with `-Z build-std=core` (and `rust-src`). The
  gate already passed that flag -- it simply had not run.

## What the verifier actually says, in the order the errors appeared

`processed 0 insns` on a 2896-instruction program is the signature of
`check_subprogs()` failing: it walks every BPF-to-BPF call target and requires each
**subprogram's** last instruction to be an exit or unconditional jump, and it runs
**before** any instruction is verified. So the error names "last insn" while the *main*
program's last instruction is correctly an exit.

Three loader defects sat behind that, each invisible until the previous was fixed:

1. **A call target ignored the symbol's offset within its section.** `.text` holds memcpy
   at 0, memmove at 0x58 and memset at 0x120; copying the whole section and aiming every
   call at its start sends `memset` to `memcpy`. The call reaches a *real* subprogram, so
   the program loads far enough to walk the CFG and reports the first instruction no path
   reaches -- memmove's, since only memset was called. `unreachable insn 2907` named a
   function nobody was looking at. Copy from `sh_offset + st_value`, not `sh_offset`.

2. **`.text.unlikely.` holds two adjacent *single-instruction* panic subprograms.** The
   first becomes an exit under the panic rewrite, so the CFG walk terminates there and the
   second is never entered; `check_cfg` demands every instruction be explored. Drop the
   second -- but scope that to the panic block. A general "drop the last instruction" rule
   truncates `memset`, an eight-instruction function whose loop then cannot be explored.
   That mistake cost a round.

3. **The trailing `exit` appended to fix error 1 became the cause of error 2.** Once every
   appended subprogram terminates in its own right, nothing falls through to a trailing
   instruction. The requirement and the old fix were one idea seen from two sides: a
   program whose last instruction is both reachable and terminal.

    last insn is not an exit or jmp   -> a subprogram, not the main program
    unreachable insn 2907            -> memset aimed at memcpy
    unreachable insn 2905            -> the trailing exit
    processed 2445 insns             -> the verifier is finally checking

**A panic subprogram that calls itself has no terminating instruction at all**, so
appending an exit *past the end of the program* can never satisfy the check. Patching only
the `goto -1` half of a `call 0x0` + `goto -0x1` pair changes nothing.

`verify-bpf.sh` **rebuilds the loader from the committed `vm/load.c` on every run**, so a
binary staged into the VM is overwritten. Know that before concluding a change had no
effect -- one experiment was void for exactly this reason.

## L0 is cut: not shipped, gate skips with a reason

L0 was **cut from shipping scope** on 2026-10-04 (`068fcf4`). The `bpf` gate now
**skips with the reason** rather than failing, prints the verifier's own words so the
evidence stays live, and fails hard under `IRONROOT_BPF_STRICT=1`. Final state:
11 PASS, 1 SKIP, "all gates passed".

`docs/bpf-loader-notes.md` is the handoff: the three loader bugs, the
instruction-count progression, the failed experiments, and the stop point. The BPF crate,
`vm/` and the loader stay in the tree, and CI has an `l0-builds` job so "it still builds"
stays true.

Reasoning worth not relitigating: L0 is an **optimisation, not a security control**. The
userspace path validates signed, unsigned-by-proof, forged and downgrade cases end to end.
A permanently red gate with an unactionable verdict is worse than a documented one -- that is
exactly how the MSYS path-mangling bug hid for days behind "object identity unproven".

## Harness traps that cost real time

- **`bind("127.0.0.1:0")` can return a port inside a Windows *excluded range*.** The TCP bind
  then fails with `os error 10013` ("access permissions"), not "in use", and **retrying cannot
  escape it** because the OS keeps offering the same reserved range. Draw test ports from a
  fixed base instead. Mutation-verified: 22 of 30 runtime tests fail with the OS-assigned port.
- **A section symbol has an empty name and `st_value` 0**, so a relocation naming `.text`
  carries no addend. That makes `bounded<1>` and `bounded<2>` indistinguishable to the loader.
  Caching "append each section once" is correct in principle and unfixable with section-symbol
  relocations -- it aims every call at offset 0 and leaves the other monomorphisation
  unreachable. Do not go down this path without per-symbol relocations.
- **`vm/verify-bpf.sh` rebuilds the loader from the committed `vm/load.c` every run**, so a
  binary staged into the VM by hand is silently overwritten.
- **Do not delete an instruction from the middle of a BPF program.** Every following relative
  jump breaks (`jump out of range`); you must fix up all later offsets.
- **The harness's own identity check can measure a stale file.** It extracted the initramfs from
  a hardcoded path while the image is written to a run-specific directory. Cross-check
  `PROG_MD5` (the object the verifier actually loaded) against the host hash before believing
  any "identity unproven" verdict.
- **Cosmetic edits can change BPF codegen.** Removing an `#[inline(always)]` that sat inside a
  doc-comment block changed `.text` from 3 functions to 7 (extra `bounded<N>` monomorphisations)
  and regressed 2445 instructions to `processed 0 insns`. Verify a BPF build after any change
  to that file, however unrelated it looks.

## Verify claims against source before writing them down

A plausible number entered a status report, got copied forward, and became a "fact" that was
never in the code: a **6,000 ms total-per-server budget** that did not exist. Only
`UPSTREAM_TIMEOUT` (1,500 ms *per attempt*) was real, with retries counted by
`MAX_REFERRALS`. It was about to be documented when a check against the source caught it.

Same shape as the codegen trap, in prose instead of instructions: something asserted early,
copied forward, and never re-derived. Check behavioural claims against the source before
documenting them, not against memory or an earlier report.

## Mutation tests: check the mutation removed the property

Replacing `match timed.await { Ok(o) => o, Err(_) => ... }` with `unwrap_or_else` left the
timeout **fully intact**, and the test passed. Replacing the whole call with a bare `.await`
was the mutation that actually removed the behaviour -- and the test then failed, reporting
`the budget did not stop the query: 30.008s`.

A mutation that leaves the property intact proves nothing. Ask what the mutant would have to
look like for the property to be gone, and check the test fails on *that*.

## Budgets: an attempt timeout is not a query timeout

`upstream_timeout` bounds one attempt; retries are counted, so the query ceiling was
`attempts x timeout`. `total_budget` (3,000 ms, `--total-budget-ms`) is the missing limit:
the pipeline runs inside `tokio::time::timeout`, and expiry gives SERVFAIL with AD clear and
nothing cached. Cancelling the future drops the in-flight `recv_from` rather than leaking a
task or socket.

Implementation note: the stages had to move into `run_pipeline` because an `async fn` body
cannot be borrowed across a timeout -- the whole loop has to live in its own future.

## Iteration and validation do reach the right servers

Checked rather than assumed, since a budget over a broken resolver treats a symptom:

- The **iterator genuinely iterates**: root -> TLD -> authoritative, 70 referral references,
  and a live 121/121 corpus against the real root servers.
- The **validator has zero `ROOT_SERVERS` references**. It uses the configured `UpstreamConfig`
  and a shared delegation cache (33 references), so DS and DNSKEY go to the right server.

## Set resource ceilings from the maximum, never the median

The `total_budget` default was a judgement call; measuring it showed the guess was wrong in the
dangerous direction. `differential-corpus` prints the cold-path distribution (first-try answers
only -- a retry's cost is harness load, not resolver latency):

    budget    p50     p90     p99    max        SERVFAILs
    3,000ms   399ms   983ms   2s     3.005s     3
    30,000ms  398ms   914ms   1s     2.087s     0

At 3 s the max lands at 3.005 s -- **the ceiling firing** -- with SERVFAILs. At 30 s the same
sweep has none and tops out at 2.087 s, the true healthy maximum. The default is now **5 s**
(~2.4x margin), mutation-verified by reverting to 3 s.

The reasoning that produced the wrong answer is worth keeping as an anti-pattern: "typical
latency is well under 100 ms, so 1.5 s is generous." A p50 says nothing about the maximum. A
resource ceiling set below the slowest *healthy* resolution converts a slow-but-working server
into a failure for every client behind it.

**Raising a ceiling is safer than lowering one.** When tuning, err upward, and re-measure.

This generalises past timeouts: any resource limit set below the observed healthy maximum
silently converts working systems into failures, and it fails *cleanly* -- a ceiling that fires
correctly looks identical to a ceiling set wrongly, in every aggregate. Guard the default with
a test asserting **both** bounds; the upper bound catches an unbounded limit, the lower catches
one that fires on healthy work. Assert against the *measurement* in the message, so the next
tuner sees where the number came from.

## Async ceilings do not cover CPU work

`tokio::time::timeout` fires only at an `.await`. A module doing sustained compute is not
interrupted mid-operation. Fine for this resolver today (DNSSEC verification is microseconds to
low milliseconds; stages are I/O-bound on DS/DNSKEY fetches), but a future compute-bound stage
needs its own bounds.

## Reading a BPF refusal

`processed 0 insns` is the signature of `check_subprogs` rejecting a **subprogram** before
instruction verification begins -- the message says "last insn" while the *main* program's last
instruction is correctly an exit. A non-zero count means the verifier is genuinely checking.

Lesson repeated twice: **the mechanism in a bug report may be right while the named culprit
is wrong.** Verify the named thing exists. One report blamed `aya-log-ebpf`'s `info!` --
the source contained no `info!` and no `aya_log_ebpf` reference at all. Reading the
disassembly of the actual object settled it in one step.

Two `load.c` facts worth keeping: a `call` immediate is a **signed instruction distance
from the calling instruction**, and code from other sections must be **copied into** the
program array with `insn_cnt` covering the appended code.

`verify-bpf.sh` rebuilds the BPF *object* from the Windows tree every run. The
*loader* that actually executes it lives inside the initramfs at
`/home/ironroot/vm/root_base/loader` and is **not** rebuilt. Days of edits to
`vm/load.c` were silently untested, and the gate kept reporting the same error the
whole time. After touching `load.c`:

```
cp vm/load.c ~/vm/ && cd ~/vm && gcc -O2 -static -o load load.c && cp load root_base/loader
```

Three bugs lived in that untested file. A `call` immediate is a **signed instruction
distance from the calling instruction**, not a byte offset and not a file offset — the
old code wrote `symbol value + section file offset`, which is short by the program's own
start offset and puts `.text.unlikely.` outside the program entirely
("call to invalid destination"). Code in another section must be **copied into** the
program array, since a BPF program is one contiguous block. And `insn_cnt` has to count
the appended code, or the destinations exist outside the submitted range and produce the
*same* error — which is what made the second bug hard to spot.

Always print the value actually submitted, not the variable that happens to be nearby.
A diagnostic reading a stale variable is worse than none: it reports a plausible number
and hides the effect of the change.

## Never test against a stale binary

A temporary debug line failed to compile, so the "fix worked" run actually exercised
the previous artifact and reported a pass. `cargo build` returning non-zero is not an
optional detail: check the exit status before believing any result from a rebuilt
binary, and prefer a mutation that must *fail* to a test that must pass -- a green test
that never ran proves nothing.

The same trap when a mutation silently does not apply: assert the mutated text differs
from the original before running anything, or the "detected" result is fiction.

## Gate wiring: a gate that cannot run must say so

The corpus sweep was added to the `GATES` array but never dispatched. It printed
`unknown gate: corpus` and the run still reported "all gates passed". A gate that
cannot run must fail, not skip quietly -- a silent skip is a green tick it did not earn.

Related, all found the hard way: `A=x B=y` on one line is a single command taking `A` as
its name, so **neither** variable is set. `--bin foo` builds only `foo`, leaving
sibling binaries absent so later checks run against a missing file -- and a missing
executable reads as a *passing* security check. `[ -f "X:/..." ]` is always false from a
POSIX shell, so Windows-form paths silently miss files that exist; and guessing the
POSIX form (`/X/...` rather than MSYS's `/mnt/x/...`) is worse than not converting,
because an unverified guess looks like a real path.

## DNSSEC verdicts: absence of evidence is not forgery

The two worst bugs were verdict-mapping errors, in **different code paths**:

| Situation | Correct | Was |
|---|---|---|
| Unsigned zone, answer has no RRSIG | Insecure | `Err` -> SERVFAIL |
| Unsigned zone, NODATA with no SOA | Insecure | `Ok(Some(Bogus))` -> SERVFAIL |

`Missing SOA record for NODATA` is only a violation in a **signed** zone. In an
unsigned zone the server is not obliged to send one. `github.com` publishes no DS, so
both its plain answers and its NODATA answers were refused -- one of the most-visited
names on the Internet.

The guard stays narrow: **zero RRSIG records anywhere** means Insecure. Anything
carrying signatures that fails to parse still fails closed. Fix one arm and check the
other -- the first fix did not cover the second.

**Mutation-test every verdict fix.** Reverting the Bogus-path fix left the suite green
until unsigned-NODATA coverage existed.

### Distinguish a broken zone from a resolver bug

A sweep flagged `go.dns.pl` as SERVFAIL. Chasing it: `dns.pl` publishes a DS but
serves no DNSKEY -- an unsigned island -- and Google and Cloudflare SERVFAIL on it
too. Ironroot answers `dns.pl` with **AD=1**, more strictly than either. Broken zone,
not a bug; SERVFAIL was right. Check whether independent references fail the same way
before "fixing" anything.

## Two gaps found by probing the header, not the code

Reading the resolver found nothing in these two places; a sweep of header fields
did. Both produced a **correct-looking response with the AD bit set**, which is
the worst kind of bug for a validating resolver: the answer looked verified.

**No opcode validation existed at all.** Every opcode was resolved as QUERY:

```
sent opcode=2 (STATUS) -> rcode=0, 701-byte answer, AD=1
sent opcode=15         -> rcode=0, 701-byte answer, AD=1
```

UPDATE is the dangerous case: a client could believe a record had been changed.
RFC 1035 4.1.1 requires NOTIMP, with the opcode **echoed** so the client can tell
which outstanding request was rejected.

**QCLASS was mixed into the cache key but never validated.** So CHAOS was
forwarded upstream as IN:

```
version.bind class=3 (CHAOS) -> rcode=0, 1031 bytes, 6 authority records, AD=1
example.com  class=3          -> the validated IN answer, verbatim
```

A recursive server answers only for IN. The lesson generalises: **a field being
present in a hash key is not the same as it being checked.**

## Unit tests cannot prove enforcement

The first opcode test called `build_response` directly. It passed. Removing the
*entire* kernel check that enforces NOTIMP left it **green**, because
`build_response` never enforced anything -- it just formats. Mutation testing
caught this immediately.

Rules that follow:
- A test of a *formatter* is not a test of a *decision*. Drive the kernel.
- If mutating the code under test does not fail the test, the test is decorative.
- **Do not truncate the captured output before deciding.** `cargo test ... | tail -5`
  cut off the `test result: FAILED` line, so three real mutations read as "not detected"
  and three load-bearing links looked untested. They were all detected. Capture the whole
  run, or grep for the marker rather than reading the tail.
- Check first whether the enforcement already existed elsewhere. The QR-set check
  was already in `Query::parse` (`NotAQuery`), so a kernel check for it was dead
  code; and asserting on the kernel instead of the parser would have passed for
  the wrong reason.
- When a test fails with a confusing number, suspect the test. `flags >> 11`
  yielded `17` (QR|rcode), not the opcode; the fix was masking `0x7800`.

## Never send more than the client asked for (RFC 6891 6.2.3)

There was no truncation at all. A client advertising 512 bytes got 1319 with TC
clear, and got the same 1319 whatever it asked for.

- Drop records **whole**, in section order, stopping at the first that does not fit.
  Keeping a later, smaller record in the gap leaves contents that do not match the
  counts.
- Never drop the question: a response whose question does not match is discarded
  (RFC 1035 4.1.1), turning a working answer into silence.
- **Cache the complete answer; fit a copy per client on the way out.** Caching the
  truncated bytes made every later client receive a partial response.
- The **cache-hit path must truncate too.** Returning cached bytes verbatim is the
  common case and no cold-cache test observes it.

### TC is bit 9 of the flags word -- byte **2**

```rust
out[2] |= 0x02;   // correct: TC
out[3] |= 0x02;   // WRONG: sets bit 1 of the low byte, inside the 4-bit RCODE
```

The wrong version turns every NOERROR into SERVFAIL, and only for responses large
enough to truncate -- so small fixtures pass and live traffic fails.

### TC cannot be derived from the final size

A truncated response *shrinks to fit* and still carries TC. So `len(msg) <= limit`
says nothing about whether TC should be set. Two of my own assertions were wrong
before the code was. Compare against the **complete** answer's size instead.

## Measuring: fresh process per configuration

Reusing one process lets the first request warm the cache, after which every later
configuration tests the cache-hit path. Three wiring mutations passed this way.
Each size needs its own process, or its own name.

To prove a mutation is load-bearing, measure directly rather than trusting a test:

```
correct code      : cached@512 -> len=456  tc=1   OK
cache-hit bypassed: cached@512 -> len=1319 tc=0   VIOLATION
```

### Two self-inflicted errors worth remembering

- `cargo fmt` reflowed `<< 16` into a multi-line form; a later `str::replace`
  silently matched nothing, so the change appeared not to take effect. **Assert the
  replacement applied.**
- Moving new functions to sit before `mod tests` deleted the entire test module. When
  reordering a file, restore from git and diff.

## Header fields that were never validated

Four fields the pipeline either ignored or checked only incidentally. All four
produced a correct-looking response with **AD=1**, which is the worst shape for a
validating resolver. Found by sweeping header fields against the running binary,
never by reading code:

| Field | Was | Now |
|---|---|---|
| OPCODE | every opcode resolved as QUERY | NOTIMP, opcode echoed |
| QCLASS | mixed into the cache key, never checked | REFUSED unless IN |
| EDNS version | read out of the OPT TTL and discarded | BADVERS |
| AXFR / IXFR | resolved as lookups -> SERVFAIL | REFUSED |

SENTFAIL vs REFUSED matters for transfers: SERVFAIL means "retry later", so a
transfer client retries forever against a server that will never comply.

## RFC 6891 OPT TTL layout

Counting from the **most significant** bit:

```
|EXTENDED RCODE (8)| VERSION (8) |DO|Z| RCODE (4)|
```

So the extended rcode is bits 31..24 and the version is bits 23..16. Writing
BADVERS into bits 16..23 puts it in the VERSION field, so a client decodes your
BADVERS as "EDNS version 16".

## Four wrong versions of one test reader

Proving the BADVERS fix took four attempts, and every failure looked like "the fix
is broken" when the fix was fine. The last reader read ANCOUNT from bytes 4..6 --
which is QDCOUNT -- so it saw ARCOUNT as 0 and returned `None` for every response,
including a correct one.

- **Dump the bytes.** Do not reason about offsets. `println!("{:02x?}", resp)`
  plus a Python one-liner resolves in seconds what an hour of reading does not.
- Header counts are QD 4..6, AN 6..8, NS 8..10, AR 10..12. Getting this wrong is
  easy to read as "the three counts after the flags".
- A helper returning `Option<u8>` where you expect a value will often be `None` for
  a reason unrelated to the fix. Check which.

## Windows reserved ports cause 1-in-4 flakes

`bind()` on a port in the OS exclusion range returns **PermissionDenied** (10013),
not `AddrInUse`:

```
cmd /c "netsh int ipv4 show excludedportrange protocol=tcp"
```

Roughly 1,700 ports are reserved for outbound use, and `bind("127.0.0.1:0")` can
hand one back. The symptom is indirect: the resolver logs
`TCP listener failed: ... 10013`, correctly keeps serving UDP, and a TCP test then
fails with a connection refusal. It reads as resolver flakiness.

- Test harnesses must retry on a port that will not bind -- **on any error**, since
  Linux reports the same collision as `AddrInUse`.
- Probe **both** transports before returning a port. A UDP-only probe picks numbers
  TCP cannot use.
- Never let a harness draw a conclusion from a single listener. `transport::run`
  treats a failed listener as non-fatal by design, which is right in production and
  wrong to assume in a test.

## Test traps in this codebase

- `ttl::for_each_section(msg, start, count, f)`: `start` is **how many records to
  skip**, not a section ordinal. Use `(msg, 0, ancount)` for answers,
  `(msg, ancount, nscount)` for authority. Passing `1` silently reads the wrong
  records with no error.
- `wire::resolve_name` returns labels **without** the root byte; decode with
  `dnsname::wire_to_name_labels`. `Question.qname` *does* include it, so
  `Name::from_wire` is right there.
- In an integration test `crate::` is the test binary, not the library. Fixtures must
  say `ironroot::wire::...`.
- `Kernel` is single-owner; `Services` is not `Clone`. Use
  `Kernel::shared_services()` to keep observing services after `transport::run`
  consumes the kernel. `Kernel` itself is now `Clone` (all `Arc` fields).
- `Kernel::handle` is async: each test thread needs its own current-thread runtime.
- `clap` reports an unknown flag on **stderr**; a shell gate checking
  `binary --flag 2>&1 | grep` must capture both or it fails a correct binary.
- In a `{ }` group, `exit` ends the whole script. Use `( ... )` for a subshell.
- Generate proptest inputs from `prop::sample::select(ALPHABET)`, not
  `any::<char>()` plus a filter: filtering to ASCII rejects ~95% of draws and
  proptest aborts with "Too many local rejects", which looks like a pass and is not.
- `MAX_TTL` clamps a 136-year TTL to one day deliberately. A property asserting
  `cached_ttl >= record_ttl` fails above 86400 and looks like a resolver bug.
- Run new suites three times. A stress test whose *final* assertion does a real
  upstream lookup will flake, because the test just exhausted that path.
- A **readiness poll must use a short timeout.** `TestServer::wait_until_ready` once
  used the same 10s read timeout as a real query inside a 10s deadline, so one
  unanswered probe ate the whole budget and the loop could not retry. It passed on
  Windows and failed on the CI runner only because startup was slower (21.5s of
  doomed waiting versus 0.5s). Hence `try_query_within(msg, timeout)`: 200ms to
  poll, 10s to ask a real question.
- **Green locally is not green on Linux.** CI runs ubuntu; the BPF toolchain lives
  in Debian WSL. Check both before pushing, and install `rustfmt`/`clippy` for the
  *stable* toolchain in WSL, not just nightly, or `cargo fmt` fails there.
- `cargo clippy` finds real defects, not just style: it caught a retry loop that
  returned on its first iteration and so could never retry, and a `u16 | u8`
  mismatch when a "redundant" cast was removed. Trust its `-- -D warnings` run.
- The BPF target's nightly rejects `fn f<N: usize>`; write `fn f<const N: usize>`.

## BPF verifier

Use `vm/bisect_bpf.py`, `vm/bisect_reads.py`, `vm/bisect_bounds.py` -- each edits,
rebuilds in WSL, reports, reverts. A round costs ~4 min and eliminates a category.

Rules, all hard-won:

- Every packet access must be a **constant offset** from one base pointer.
- **Integer** bounds checks (`offset < len`) prove nothing. Compare a derived
  pointer against `pkt_end`. Arithmetic *on* `pkt_end` is forbidden; derive one
  pointer and add a constant to it.
- Any `[u8; N]` with a fill byte is a **memset**, pulling compiler-builtins into
  `.text`, which aya concatenates into the program. Use `MaybeUninit`.
- Prefer a **const ladder** (64/32/16/8/4/2/1): a 13-byte name is shorter than a
  64-byte window and a 255-byte name overflows the 512-byte stack.
- `read_u8`, `read_be16`, `write_be16` must all go through `read_span`/`write_span`.
  A bare `get` or two separate byte reads fails verification while passing every
  host test.

Harness: `~/vm` in Debian WSL, `vm/verify-bpf.sh` MD5-checks the object against the
copy embedded in the initramfs. Never trust a verdict without that match.

**Status: not accepted, but the cause is now pinned.** Staging the QNAME into a
stack buffer *does* clear the verifier's packet-access complaint. The blocker
moved to a different wall, and it is a specific one:

> The write into the staging buffer must leave **no panic call**. aya concatenates
> `.text.unlikely` into the program, so any panic the compiler cannot prove
> unreachable becomes a trailing `call` the loader cannot resolve -- and the
> symptom is "last insn is not an exit or jmp", which says nothing about panics.

Five shapes tried, one failure each:

| shape | leaves |
|---|---|
| `copy_from_slice` at a runtime length | `memmove` |
| `buf[i + k]` | a bounds-check panic |
| `buf.get_mut(a..b)` | a range-check panic |
| `buf.as_mut_ptr().add(i)` cast to `[u8; N]` | an alignment panic |
| nested `[[u8; BLOCK]; N]` | two index checks |

The nested array ships (correct, tested, two panics left). Next: a fixed-width
`copy_from_slice::<$n>` between two arrays of the same compile-time width -- no
runtime length anywhere in it, so the remaining checks are over constants.

Also learned the hard way:

- The ladder's width must come from what the **packet** offers, not the buffer's
  remaining space. Backwards stages nothing for any query shorter than STAGE.
- A name longer than one block crosses the boundary; dropping the spilled bytes
  silently changes the hashed name, so the key stops matching the userspace
  formula. Mutation testing caught this one.
- STAGE must fit the 512-byte frame alongside `build_reply`'s buffers: 64 works,
  128 does not. A longer name than STAGE is an L0 miss, which is correct.
- The staged length is a **lower bound** when the packet is short; the walk must
  stop there rather than read uninitialised stack.
- `rm` the panic and the check becomes redundant: the compression-pointer guard is
  subsumed by `label_len > 63`. Kept and documented as defence in depth with the
  mutation result, rather than implied to be load-bearing.

`vm/bisect_*.py` are the tools that got here; a round is ~4 min.

## Hermetic end-to-end: per-resolver config, and what it unblocked

A mock root and a mock authority must be **different sockets** -- the resolver
expects a referral from one and a final answer from the other.

`UpstreamConfig` carries the root set and the delegation port on the resolver
itself, reached via `RuntimeConfig` and `Services`. It must be per-resolver, not
process-global: env vars were tried twice and both failed -- read once per process
pinned every test to the first test's root (a silent false pass), read live each
test overwrote the previous mock (wholesale timeout).

**The root's port and the delegation port are different fields.** A glue record
carries only an IP, so the delegation port is ours to choose; the root's port is
part of a real address we were handed. Collapsing them points root queries at the
authority's socket.

Defects that surfaced only once the suite could run:

- `TestServer::_mock` was `Option`, so a second `with_mock` silently dropped the
  first -- the mock authority, whose socket closed on drop. Indistinguishable from
  a resolver that cannot reach its delegated server. It is a `Vec` now.
- **A negative answer was treated as a referral.** Empty answer + populated
  authority is both shapes; the code discriminated on section *counts*. NXDOMAIN
  carries an SOA there (RFC 2308 2.1), so it went back to the root and died as a
  referral loop. Check for actual NS records.
- `resolve_name` returns **labels, not encoded bytes**. Reading qtype at
  `12 + resolve_name(..).len()` silently decodes 0.
- A mock that echoes a hardcoded qtype gets its referral rejected as a mismatch --
  correctly. Echo the query's type.

Still ignored, each with its reason: two need an NSEC proof of DS absence to
establish an insecure delegation (needs a signed parent zone); one needs
per-query rcodes from the mock. The resolver is *right* in the first two --
unable to distinguish unsigned from forged, it says bogus, which is correct.

A mock **can** assert a secure verdict. It was previously assumed it could not,
which is why the suite could only ever pin "unverifiable data is withheld" and
never "signed and valid" -- the distinction those tests exist to draw. See below.

### Mocks must serve TCP, and truncation must be transport-specific

RFC 1035 §4.2.1 makes the resolver retry a truncated UDP response over TCP, so a
UDP-only mock cannot test it at all. Serve UDP and TC

…(truncated)
