Spec To Code Fidelity

Spec-to-Code Fidelity

Status: v0.1 baseline draft — derived from the case catalog and axis definitions; awaiting trace evidence from Fable 5.1 case runs.

When to apply

• Implementing a protocol or format from an RFC, W3C/WHATWG spec, or vendor format document (HTTP/1.1, WebSocket framing, GGUF, JWT, WAI-ARIA).
• Translating equations from a paper into code (attention math, quantization schemes, PPO loss, numerical integrators).
• Writing a parser/serializer for a binary or wire format where a single off-by-one silently corrupts everything downstream.
• Porting a reference implementation to another language while claiming behavioral equivalence.
• Any task where the acceptance criterion is "conforms to the document" rather than "seems to work."
• Debugging interop failures where your implementation and a peer disagree about what the spec says.

Core loop

1. Acquire the authoritative text before writing code. Get the actual RFC/paper/spec section into context — not your memory of it. If the exact document version matters (RFC obsoleted-by chains, paper v1 vs v2 on arXiv), pin the version and record it in a comment at the top of the implementation.
2. Decompose the spec into a numbered clause inventory. Walk the document and list every normative statement (MUST/SHALL/SHOULD, every equation, every field of every struct/frame, every state transition). Give each clause a stable ID (e.g. §5.2-3, Eq.4). This inventory is your requirements list and later your test matrix.
3. Adopt the spec's vocabulary in code. Name identifiers, types, and constants exactly as the spec names them (opcode, masking_key, d_model, scale/zero_point), even when the spec's names feel awkward. Cite clause IDs in comments next to the code that implements them. A reader holding the spec must be able to diff document against code by eye.
4. Implement clause-by-clause, marking coverage. For each clause: implement it, tag it [done], [deviates: reason], or [ambiguous: interpretation chosen]. Never let a clause silently fall into "probably handled." Deviations and ambiguity resolutions go in a SPEC_NOTES section (file header or separate doc) — this is the artifact interop debuggers will need.
5. Extract every worked example and test vector from the spec first. RFCs carry example exchanges; papers carry example tables and reported figures; format specs carry sample hex dumps. Turn each into an executable test before or during implementation — these are the only tests whose expected values you didn't invent. Concrete artifacts: a vectors/ fixture directory holding the bytes or values verbatim from the document, and one test per vector named after its clause ID (test_5_7_masked_hello) so a failing test names the clause it violates.
6. Cross-check against a reference implementation or oracle where one exists. Run the canonical implementation (curl for HTTP, jwt.io vectors, the paper authors' repo, llama.cpp for GGUF) on the same inputs and diff outputs byte-for-byte or number-for-number. Make the diff a committed script (scripts/oracle_diff.sh), not a one-off shell session — you will re-run it after every substantive change and once more at the final audit. Where no oracle exists, hand-compute at least one small case on paper and commit it as a fixture, with the derivation recorded in a comment next to the expected values.
7. Validate numerics against known values before trusting downstream behavior. For math-heavy specs: check intermediate quantities (attention weights sum to 1, quantize→dequantize round-trip error within the scheme's stated bound, loss at init ≈ ln(vocab_size)) against analytically known values. Encode these as assertions or a standalone sanity-check entry point that prints each invariant with its observed value, so the check re-runs for free on every future change. A converging loss curve or a "working" demo is not evidence the math is right — many wrong implementations still converge or interoperate by accident.
8. Audit clause coverage before declaring done. Sweep the clause inventory: every clause is [done], [deviates], or [ambiguous] with a test or a note. List untested clauses explicitly in the final report. "All tests pass" is meaningless if the tests cover 30 of 80 clauses.
9. Re-read the spec once more after implementation. A final pass with working code in hand catches misreadings that were invisible on first read — you now know which sentences you skimmed. Budget this pass; it is where the subtle bugs (network byte order, inclusive vs exclusive ranges, 0- vs 1-indexed equations) surface. Exit: every clause accounted for, spec vectors green, oracle diff (or hand-computed fixture) green.

The clause inventory artifact

Maintain the inventory as a literal artifact — a SPEC_NOTES block in the file header or a sibling doc, not working memory:

SPEC: RFC 6455 (no errata relevant to §5.2), fetched <date>
CLAUSES:
  §5.2-1  FIN/RSV/opcode bit layout            [done]  test_5_2_bits
  §5.2-2  7/16/64-bit payload length encoding  [done]  test_5_2_len16, test_5_2_len64
  §5.3-1  masking algorithm: j = i MOD 4       [done]  test_5_7_masked_hello
  §5.5-1  control frames ≤125B, unfragmented   [deviates: also reject close frames
          whose reason is invalid UTF-8 — stricter than the spec requires]
  §5.5.1  reserved close status codes          [ambiguous: spec silent on 1015 when
          received; chose: treat as protocol error, matching browser behavior]
UNTESTED: §5.5.1 reserved codes — reported explicitly, low interop risk

Rules for the artifact:

• One row per clause; the coverage tag and its test name (or note) live on the same line, so grep '\[deviates' produces the complete deviation report in one command.
• Rows are retagged, never deleted, once implementation starts — a deleted row is a clause you forgot, not a clause you resolved.
• The audit in step 8 is a mechanical sweep of this block, not a re-read of the code; if the sweep requires judgment, the tags were written too loosely.

Heuristics

• If the spec gives a worked example, implement enough to reproduce that exact example before anything else — it is the cheapest full-pipeline test you will ever get.
• MUST/SHALL clauses each get a test; SHOULD clauses each get a test or a one-line documented decision to skip.
• When the spec is ambiguous, do not average interpretations: pick one, write down why, and check what the dominant reference implementation does — interop reality usually trumps a literal reading.
• Hand-compute vs oracle rule: if a runnable oracle exists and executes in under a minute, diff against it at every milestone; if the only oracle is expensive (a training run, a remote peer, special hardware), hand-compute one case small enough to verify on paper (≤ ~20 arithmetic steps) and pin it as a fixture before building anything on top.
• When porting a reference implementation, port its test suite before writing your own tests; a port that cannot pass upstream's tests is not equivalent, whatever your own tests say.
• Never copy a magic constant from another implementation without locating it in the spec; if it is not in the document, it is a deviation and belongs in SPEC_NOTES with its source named.
• Any identifier renamed away from spec terminology costs you a mapping table; if you must rename, keep an explicit spec name → code name table in SPEC_NOTES.
• In binary formats, verify field offsets with a hex dump of a real file before writing the parser loop — one wrong struct size shifts every subsequent field.
• For equations: transcribe the formula verbatim into a comment above the code, including index ranges and summation bounds; most math bugs are bound/index errors, not operator errors.
• Numeric tolerance for cross-checks: bitwise-exact for integer/serialization specs; for floating point, assert against the spec's reported figures at their printed precision, and treat >1 ULP-scale drift in a supposedly deterministic step as a bug until explained.
• If your output disagrees with the oracle, assume you are wrong first; only claim an oracle bug after you can quote the exact clause the oracle violates.
• Endianness, sign-extension, padding, and inclusive/exclusive boundaries account for a disproportionate share of spec bugs — grep your own diff for byte-order and range operations during the final re-read.
• Budget the final re-read at roughly 10% of implementation time; skipping it trades a fixed small cost for open-ended interop debugging later.

Anti-patterns

• Implementing from memory of the spec. Training-data recall of an RFC blends versions and errata. → Corrective: load the actual text; quote clauses into comments as you implement them.
• Paraphrase drift in identifiers. Renaming nonce to randomToken "for clarity" breaks the eye-diff between doc and code and hides misunderstandings. → Corrective: keep spec terms verbatim; put clarity in comments, not names.
• Testing only self-invented cases. If you wrote both the code and the expected values, a shared misreading passes green. → Corrective: prioritize spec-provided vectors and oracle diffs; your own cases are supplementary.
• Validating math by downstream behavior. "The model trains" or "the peer accepts my handshake" can be true of subtly wrong implementations. → Corrective: assert intermediate values against known constants before running end-to-end.
• Silently resolving ambiguity. Choosing an interpretation without recording it makes the eventual interop bug undiagnosable. → Corrective: every ambiguity gets a SPEC_NOTES entry with the chosen reading and the alternative.
• Declaring done on green tests without a clause audit. Coverage of the test suite is not coverage of the spec. → Corrective: sweep the numbered clause inventory and report the untested remainder explicitly.

Worked example

Illustrative construction, not a recorded run.

Task: implement WebSocket frame parsing (RFC 6455 §5).

Agent fetches RFC 6455 §5 and checks the errata index → pins "RFC 6455, no
errata relevant to §5.2" in the file header, because implementing from memory
blends this frame layout with other protocols' habits.
Agent walks §5.2-§5.5 → emits a 14-clause inventory: FIN/RSV/opcode bits
(§5.2), payload length 7/16/64-bit encoding (§5.2), MASK bit + masking
algorithm (§5.3), control frames ≤125 bytes and never fragmented (§5.5),
close status codes (§5.5.1).
Agent declares struct Frame { fin, rsv1..3, opcode, mask, payload_len,
masking_key, payload } → field names verbatim from the spec figure, each
commented with its clause ID, because the doc-to-code eye-diff must work.
Agent extracts §5.7's worked examples ("Hello" unmasked = 0x81 0x05 0x48...,
masked "Hello", 256-byte binary) into vectors/ and writes one test per vector
before the decoder exists → the only expected values the author didn't invent.
First run: unmasked "Hello" passes; masked "Hello" decodes to garbage.
Agent re-reads §5.3 instead of debugging the code first → because when code
and spec disagree, the cheapest probe is the four lines of normative text.
Observation: §5.3 indexes the key as j = i MOD 4; the code applied the key
bytes in reverse order — a memory-of-spec habit from a different protocol.
Fix → all spec vectors green.
Agent cross-checks the oracle: pipes 50 random payloads from a real client
(websocat) through the parser and byte-diffs against the library the client
trusts → zero diffs; the diff script is committed for re-use at audit time.
Clause audit: 12/14 tested. §5.4 fragmentation reassembly and §5.5.1 reserved
status codes are [done, untested] → fragmentation is a MUST, so two tests are
added; reserved codes stay untested and are reported explicitly.
Final re-read of §5.2 with working code in hand → 16-bit extended length
("in network byte order") is handled, but the 64-bit branch had a host-order
read on one path. Fixed before any peer ever sent a frame that large.
Exit: 14/14 clauses accounted for; spec vectors, oracle diff, and clause
audit all green.

Trace evidence