Fuzzing An API Surface

Fuzzing an API surface

Use this skill when you want to systematically hammer a surface that decodes input you do not control, and have every real failure become a named regression test rather than a transient console reproduction. Evener is the proving ground; the portable core (fuzz/promoter, fuzz/schemagen) carries unchanged to any project, so the methodology below is project-agnostic and the evener targets are worked examples.

Read docs/research/api-fuzzing-toolkit.md (§3 oracle soft spots, §5 the promote discipline, §8 the portable split) and docs/design/fuzzing-toolkit-design.md before extending the toolkit.

When to reach for this

A surface is fuzz-ready when it has:

• A single dispatch choke point — one function every input flows through (tool.Registry.ExecuteCall, Router.Dispatch, WebServer.Handler()). One harness then covers the whole surface by varying which operation it drives.
• A machine-readable schema — per-operation JSON Schema, reflected param types, OpenAPI, protobuf. Schema-driven generation reaches deep validator paths raw bytes never hit (evener hands us tool Definition.Parameters and the appwire.Methods catalog for free).
• A decode / parse seam — a byte-level frame/stream parser (llm.ParseSSE, appwire.Message.UnmarshalJSON). These are the cheapest, highest-value targets: a Go testing.F byte fuzzer auto-promotes its own crashers.
• A metamorphic relationship — two ways to feed the same logical input that must agree (re-chunked reads, reordered/whitespace-padded frames, blocking vs timeout code paths). The relationship is the oracle.

Do NOT reach for this when the seam's only effect is to execute something dangerous (see Safety bounds), when no determinism is achievable offline (the real behavior needs a live LLM/network/subprocess), or when there is no oracle richer than "it returned" — a fuzzer with only a floor oracle on a trivial surface buys little.

Core rules (non-negotiable)

• Safety bound: never let a fuzzer EXECUTE a dangerous handler. Shell (bash -c), network fetch, file read/write under attacker-controlled paths, subprocess/agent spawn — a temp dir does NOT sandbox these. Fuzz the **decode
  • validate boundary** instead, or restrict execution to a vetted allowlist of genuinely sandboxable operations. Evener's FuzzToolArgsValidate drives decode+Schema.Validate and never calls a tool handler; FuzzWebHandler allowlists GET-only, non-mutating, non-networked routes and excludes spawn, shell, and provider-probe endpoints.
• Determinism is non-negotiable. Seed the generator; run offline (no real LLM, network, or subprocess); a flaky target — or a promoter that could promote a flake — is a defect, not a nuisance.
• "No panic" is the floor, never the only oracle. Invest in semantic oracles; the real bugs live there.
• A thin-but-real model beats a broad fake one. Model only what you can drive deterministically against the real seam, and report the boundary you could not reach. Never fake the unreachable part and assert on the mock — that tests mocked behavior. TestRouterSeqFuzz models exactly the transport initialize-gate it can reach offline and states in its doc comment that turn/job status lifecycle needs a live agent and is out of scope.

Procedure: pick a surface and seam (research §8, 7 steps)

1. Pick a surface and its seam. Find the single choke point all input flows through. That is what the harness drives.
2. Find or derive the schema. Point generation at whatever machine-readable structure exists (Definition.Parameters, reflected appwire.Methods, OpenAPI). No schema → fall back to byte fuzzing.
3. Write the thinnest testing.F first and harvest the free auto-promotion: a single []byte or typed-tuple target whose crashers Go saves to testdata/fuzz/ and re-runs forever. This is ~60–200 LoC and finds the crash-class bugs before you build anything else.
4. Add oracles beyond "no panic" (next section). This is where valid-but-wrong bugs surface.
5. For stateful surfaces, model the state machine and drive it with property-based testing (rapid). Keep the model declarative — evener's liveOpTable is a transition table, the reusable artifact; the rapid machine just draws op sequences.
6. Wire the promoter with flake-guard before promote so non-testing.F failures (rapid sequences, HTTP op-sequences) become permanent.
7. Gate it in CI: seed corpus + saved crashers under -short on every run (make fuzz / make test); the unbounded coverage-guided search runs nightly (make fuzz-nightly, 60s/target).

The oracle taxonomy

The portable tags are promoter.OracleTag: Panic, Invariant, ErrorShape, Wedge, HTTP5xx, PathEscape. How to spot which fits, each tied to a real evener target:

• Floor — never panic (Panic). The seam crashes on some input. Always present; never sufficient alone. Live panic-hunt wherever validation is not recover()-wrapped at runtime — e.g. Schema.Validate in FuzzToolArgsValidate, and Router.Dispatch (which propagates handler panics) in TestRouterSeqFuzz.
• Metamorphic (Invariant). Two equivalent inputs must produce the same output. Spot it when one logical input has more than one delivery shape: re-chunked reads or blocking-vs-timeout paths (FuzzParseSSE), one-byte-at-a-time SSE accumulation (FuzzOpenAIResponsesMetamorphic), decode→encode→decode fixed point (FuzzMessageDecode, FuzzMethodParams).
• Divergence (ErrorShape). A validator says clean but the handler disagrees, or vice versa. Spot it when validation and use are separate steps: TestToolArgsSchemaFuzz flags a schema-VALID generated value the real validator rejects — generator and schema disagree.
• Invariant / monotonicity (Invariant). A modeled property must hold across a sequence. Spot it on stateful surfaces with an ordering contract: TestRouterSeqFuzz asserts the connection's initialize-gate is monotonic (fresh → initialized, never reverts).
• Never 5xx (HTTP5xx). Bad input may legitimately 4xx, but a 500 from your own handler is a fault. Spot it on HTTP surfaces: FuzzWebHandler (it excludes the bare stdlib http.FileServer, whose fs.ErrInvalid→500 is documented net/http behavior, not handler logic).
• Path escape (PathEscape). Attacker-controlled path/query escapes its intended root. Spot it on any file/path resolver: FuzzWebHandler plants a secret one level above the session cwd and fails if any response body contains it.
• Never wedge (Wedge). A call that should return makes no progress. Spot it on synchronous seams that could hang: TestRouterSeqFuzz runs each dispatch under a bounded context in a goroutine and reports a wedge on timeout.
• Internal invariant (Panic, under -tags evenerfuzz). A load-bearing assumption asserted inside production code with invariant.Hold() so a logic bug trips at its origin, not at a distant surface. Zero-cost in a normal build; panics under -tags evenerfuzz (the fuzz build), so it surfaces via the Panic oracle. Spot it wherever a subsystem relies on a property a reader assumes — e.g. a folded job status never leaves a terminal state; an emitted item carries its turn id. Conditions must be side-effect-free; verify the invariant is TRUE and reached before trusting it.
• Differential (Invariant/ErrorShape). Two code paths that must compute the same thing, driven from one input and compared modulo an allow-list. The strongest class — it found both real decoder bugs this codebase has caught. Spot it wherever a value is produced two ways: per-provider streaming vs non-streaming decode (FuzzStreamVsNonStreamDifferential), one logical response across all providers (FuzzCrossProviderDifferential), a decoder vs a committed snapshot of its own output (appwire/golden_test.go), or independent readers of one format (FuzzTranscriptReadersAgree). A divergence outside the allow-list is a real bug — never widen the allow-list to silence it.

Generation technique — structure-aware inputs. Independent of which oracle you pick: if the surface has a rich grammar, most raw-byte execs die at the first parse. Drive the target from a deterministic byte-Source generator (fuzz/schemagen) that emits valid-but-adversarial inputs (the Fuzz…Structured targets), so coverage-guided search reaches deep logic. It typically lifts input acceptance from ~0% to ~90%+.

The portable core

Two packages in the primeradiant.com/evener/fuzz module are the travelling tooling. Nothing in them imports the project under test — fuzz/go.mod declares no evener dependency, so the module will not build if that boundary is violated. That structural guarantee is the portability test.

schemagen.FromJSONSchema(schema map[string]any) *rapid.Generator[any] walks the JSON Schema subset a surface actually uses (type/properties/required/ enum/additionalProperties/items) and yields both schema-Valid and schema-Adjacent (wrong-type, missing-required, out-of-enum, extra-when-closed) values. Use schemagen.Generator(schema, mode) when an oracle needs known-valid input.

promoter.Adapter is the four-hook seam a project implements; everything else (flake-guard loop, bucket store, emitter, commit) is generic:

type Adapter interface {
    Minimize(Failure) Failure                                    // usually a passthrough
    Signature(Failure) Signature                                 // dedup key
    Replay(context.Context, Failure) (failed bool, sameSignature bool)
    Emit(Failure) (path string, err error)
}

A project implements it by carrying enough state to deterministically reproduce a failure. toolArgsAdapter holds each tool's compiled validator; seqAdapter holds the transition table + handler registrar. Replay re-runs the same oracle function that found the failure (toolArgsFailure / seqOracleRun), so the live property and the replay classify identically. Emit calls promoter.WriteGoTest(dir, promoter.GoTest{...}) with an adapter-rendered ReplayBody that decodes the minimized artifact and asserts the bug is fixed.

promoter.New(adapter, store, quarantiner, K) then runs Promote.

The flake-guard / dedup rubric (the load-bearing discipline)

Promoter.Promote is the whole point. In order:

1. Dedup first. store.Has(sig) against the committed BucketStore (a JSON file mapping Signature → test path). A known bug returns AlreadyKnown and is never re-promoted.
2. Flake-guard. Replay the minimized artifact K times (default 5). Promote ONLY if it fails all K with the same signature. Any non-reproduction or signature shift → Quarantined: logged via the Quarantiner, never written as a test. This is the rule that keeps the gate trustworthy; budget it generously on timing-sensitive seams.
3. Emit-only by default. A surviving failure writes a TestRegression_* file with a provenance header (// Code generated by fuzz/promoter ... DO NOT EDIT) and records its bucket. Promotion into the tree is the human/opt-in step — review the diff before committing.

Signatures must distinguish distinct bugs: Signature{Oracle, Key} where for Panic the Key is the top-N normalized stack frames, and for semantic oracles (Invariant/ErrorShape) the adapter folds the named invariant / Detail into Key. Collapse them and distinct bugs silently dedup as already-known.

Hard-won lessons

• rapid owns its shrink loop. pgregory.net/rapid reports failures through testing.T and Goexits during shrinking; it does not hand you the minimized artifact as a return value. Capture the failing artifact into a closure variable inside rapid.Check, and promote it in t.Cleanup (which runs during the Goexit unwind). Adapter.Minimize is then a passthrough — rapid already minimized. Both TestToolArgsSchemaFuzz and TestRouterSeqFuzz use this captured *Failure + cleanup pattern.
• Liberal-in / conservative-out — the fix location matters. FuzzMessageDecode found a real codec asymmetry: the decoder accepted an id-less response/error frame the encoder then re-rendered unreadable, breaking the round-trip fixed point. The fix (commit 6c7f4eac) was on the producer side — Response.MarshalJSON/ErrorResponse.MarshalJSON omit the empty id — keeping the decoder liberal. On a transport, tightening a decoder turns a tolerable frame into a connection-fatal error; tighten the producer instead.
• Model only what you can drive for real. When the real status lifecycle needs a live LLM, model the transport/dispatch layer you can reach offline and report the boundary in the target's doc comment. Do not fabricate the unreachable part.

Per-language portability

The loop, the triage/promotion discipline, schema-driven generation, and the oracle taxonomy generalize. Only the engine, corpus format, and emitter syntax are per-ecosystem, all behind the same Adapter:

• Go: testing.F (coverage-guided, free crasher auto-promotion) for byte/tuple targets; pgregory.net/rapid for stateful/structured PBT. Emitter writes a *_test.go; corpus is f.Add seeds + testdata/fuzz/.
• JS: fast-check (property/model-based) for a renderer/protocol surface; its emitter writes a test-*.js.
• Python: hypothesis (+ schemathesis for HTTP); its emitter writes a test_*.py.

Cross-module wiring (Go specifics)

A local v0.0.0 workspace module is not go get-able. To consume fuzz/promoter and fuzz/schemagen from another module in the same go.work:

1. Add require primeradiant.com/evener/fuzz v0.0.0 (and pgregory.net/rapid) to the consumer's go.mod.
2. Add replace primeradiant.com/evener/fuzz v0.0.0 => ./fuzz in go.work, then go work sync.
3. Seed third-party checksums the workspace replace hides: GOFLAGS=-mod=mod GOWORK=off go mod download pgregory.net/rapid.

Add fuzz to GO_MODULES so its tests are gated. make fuzz runs every Fuzz* target's seed corpus + saved crashers per module as deterministic tests (no -fuzz flag = no random search); make fuzz-nightly runs the bounded search.