Hunting failing inputs
A trap is an operation with a precondition the input can break: an index
in range, a slice end within the length, a non-zero divisor, a sum that must
not wrap.
Not for
| Situation |
Use instead |
| Races, deadlocks, async ordering |
race detector or stress test |
| Whole programs, I/O, network, UI |
fuzzing plus sanitizers; extract the pure core first |
| Wrong results no precondition expresses |
property tests against a reference, or reasoning-semiformally |
| Whether a test can go red |
gating |
| Validating or minifying WGSL |
compiling-wgsl |
Stop and report a trap as unmodelled (never as safe) when its model needs
more than about 15 unrolled loop iterations, needs heap state that a single
input cannot build, or returns unknown twice.
Setup
python3 -m pip install z3-solver tree-sitter-language-pack
SKILL=/mnt/skills/user/hunting-failing-inputs
python3 $SKILL/scripts/selftest.py # optional: SKIP means a missing toolchain, not a pass
Procedure
List the traps.
python3 $SKILL/scripts/list_traps.py FILE... [--func NAME] prints
file:line:col kind on-failure func source. on-failure is raises,
silent or UB (C, which is silent without sanitizers). Model the silent
and UB traps first. Strike any trap that an earlier guard already makes
safe, and in JS strike object lookups. For Go, use bce_targets.py
instead (see references/go.md).
Read references/<lang>.md for the target's language: python,
javascript, c, wgsl or go. Each one has the failure semantics, the
predicate to model each trap with, the strict harness, and a runnable
example in examples/<lang>/.
Model one pure function. Include the guards its callers enforce as
constraints. Without them, Z3 invents inputs the program can never produce.
Write the model from scripts/z3traps.py, copying the nearest
examples/<lang>/model*.py:
import sys; sys.path.insert(0, f'{SKILL}/scripts')
from z3traps import *
n, base = harness(max_len=64)
b0 = byte_at('b0')
base += [n >= 2] # every early return the path got past
solve_traps(base, {'f.py:10 b[k:k+2]': slice_truncates(k, k + 2, n)}, show=(n, b0))
- Do arithmetic at the width the language uses, and
widen exactly where
the source converts. Python ints never wrap. C narrows on a store or a
cast, JS |0 truncates to 32 bits, and WGSL u32 and Go uint8 wrap.
- Write each guard the way the source writes it, including its overflow.
A guard written the way it was meant models the fixed code.
- Use the precondition the runtime actually checks. Go checks slices
against
cap. Python wraps silently only for -len <= i < 0.
- Use
in_order=True when the traps are on one straight-line path, and
leave it off when they are on different branches.
- For loops, try concrete lengths
L = 1, 2, … with symbolic bytes, and
stop at the first sat.
- z3py's
< and >= on bit-vectors are signed. For unsigned values,
widen first or use ULT/UGE.
Replay each sat inside the strict harness. literal(bytes, lang)
renders the input for a test. Use the harness for the language:
| language |
strict harness |
| Python |
parse(strict(data)), scripts/strict.py |
| JS / TS |
decode(strict(u8)), scripts/strict.mjs |
| C |
-fsanitize=address,undefined -fno-sanitize-recover=all, input in an exact-size malloc |
| WGSL |
scripts/wgsl_guard.py counts bad indexes in an extra _oob buffer |
| Go |
d = d[:len(d):len(d)] |
Then read the result:
| Z3 |
replay |
meaning |
sat |
fails at the trap's line |
A bug. Report it with the reproducer. |
sat |
fails at an earlier line |
Re-solve with in_order=True. |
sat |
doesn't fail, or fails differently |
The model is wrong. Fix it, and never report this as a bug. |
unsat |
— |
"None within len ≤ N", not "safe". |
unknown |
— |
Simplify the model. Never count it as unsat. |
Fix, and keep the harness. Patch the code, update the model and expect
unsat, and keep the reproducer as a regression test. Add the strict
harness to the project's existing fuzz targets and property tests.
Verification
Every trap you didn't strike ends as one of:
- a reproducer that fails at that trap inside the strict harness
unsat with a stated bound, plus a negative control that went sat
- unmodelled, with the reason
For the negative control, remove one guard or widen one field and expect
sat. If the model stays unsat, it is over-constrained and its unsat
means nothing.
Failure modes
- A
sat input doesn't fail as predicted. The model's precondition is
wrong. On 2026-11-13, a Python model treated every negative index as a
silent wrap, and its input b'\x00\xc0' raised IndexError instead.
- A later trap's
sat input fails at an earlier line. Use
in_order=True. On 2026-11-13, the Go line-16 input panicked at line 14.
- Every trap is
unsat, including the buggy ones. A guard was modelled
with wide arithmetic where the source uses narrow. Run the negative control.
- The strict harness passes, but Z3 says
sat. The wrapper covers only
the object you wrapped. Buffers the function builds itself are unwrapped.
- Random testing finds nothing, even when strict. Check the random source.
On 2026-11-13, a JS LCG that lost precision past 2^53 never hit the bug.
- clang fails to link ASan or libFuzzer.
libclang-rt-<N>-dev is
missing. Install it, or use gcc.
1---2name: hunting-failing-inputs3description: Finds inputs that make a parser, decoder or WGSL compute shader index out of bounds or overflow, using Z3. Use for "can this crash", "find an input that breaks this", or when fuzzing found nothing on code that reads untrusted bytes.4metadata:5 source: https://skillmd.com/skills/oaustegard/hunting-failing-inputs6 version: 0.3.17---89# Hunting failing inputs1011A **trap** is an operation with a precondition the input can break: an index12in range, a slice end within the length, a non-zero divisor, a sum that must13not wrap.1415## Not for1617| Situation | Use instead |18|---|---|19| Races, deadlocks, async ordering | race detector or stress test |20| Whole programs, I/O, network, UI | fuzzing plus sanitizers; extract the pure core first |21| Wrong results no precondition expresses | property tests against a reference, or `reasoning-semiformally` |22| Whether a test can go red | `gating` |23| Validating or minifying WGSL | `compiling-wgsl` |2425Stop and report a trap as **unmodelled** (never as safe) when its model needs26more than about 15 unrolled loop iterations, needs heap state that a single27input cannot build, or returns `unknown` twice.2829## Setup3031```bash32python3 -m pip install z3-solver tree-sitter-language-pack33SKILL=/mnt/skills/user/hunting-failing-inputs34python3 $SKILL/scripts/selftest.py # optional: SKIP means a missing toolchain, not a pass35```3637## Procedure38391. **List the traps.**40 `python3 $SKILL/scripts/list_traps.py FILE... [--func NAME]` prints41 `file:line:col kind on-failure func source`. `on-failure` is `raises`,42 `silent` or `UB` (C, which is silent without sanitizers). Model the `silent`43 and `UB` traps first. Strike any trap that an earlier guard already makes44 safe, and in JS strike object lookups. For Go, use `bce_targets.py`45 instead (see `references/go.md`).46472. **Read `references/<lang>.md`** for the target's language: `python`,48 `javascript`, `c`, `wgsl` or `go`. Each one has the failure semantics, the49 predicate to model each trap with, the strict harness, and a runnable50 example in `examples/<lang>/`.51523. **Model one pure function.** Include the guards its callers enforce as53 constraints. Without them, Z3 invents inputs the program can never produce.54554. **Write the model** from `scripts/z3traps.py`, copying the nearest56 `examples/<lang>/model*.py`:5758 ```python59 import sys; sys.path.insert(0, f'{SKILL}/scripts')60 from z3traps import *61 n, base = harness(max_len=64)62 b0 = byte_at('b0')63 base += [n >= 2] # every early return the path got past64 solve_traps(base, {'f.py:10 b[k:k+2]': slice_truncates(k, k + 2, n)}, show=(n, b0))65 ```6667 - Do arithmetic at the width the language uses, and `widen` exactly where68 the source converts. Python ints never wrap. C narrows on a store or a69 cast, JS `|0` truncates to 32 bits, and WGSL `u32` and Go `uint8` wrap.70 - Write each guard the way the source writes it, including its overflow.71 A guard written the way it was meant models the fixed code.72 - Use the precondition the runtime actually checks. Go checks slices73 against `cap`. Python wraps silently only for `-len <= i < 0`.74 - Use `in_order=True` when the traps are on one straight-line path, and75 leave it off when they are on different branches.76 - For loops, try concrete lengths `L = 1, 2, …` with symbolic bytes, and77 stop at the first `sat`.78 - z3py's `<` and `>=` on bit-vectors are signed. For unsigned values,79 widen first or use `ULT`/`UGE`.80815. **Replay each `sat` inside the strict harness.** `literal(bytes, lang)`82 renders the input for a test. Use the harness for the language:8384 | language | strict harness |85 |---|---|86 | Python | `parse(strict(data))`, `scripts/strict.py` |87 | JS / TS | `decode(strict(u8))`, `scripts/strict.mjs` |88 | C | `-fsanitize=address,undefined -fno-sanitize-recover=all`, input in an exact-size `malloc` |89 | WGSL | `scripts/wgsl_guard.py` counts bad indexes in an extra `_oob` buffer |90 | Go | `d = d[:len(d):len(d)]` |9192 Then read the result:9394 | Z3 | replay | meaning |95 |---|---|---|96 | `sat` | fails at the trap's line | A bug. Report it with the reproducer. |97 | `sat` | fails at an earlier line | Re-solve with `in_order=True`. |98 | `sat` | doesn't fail, or fails differently | The model is wrong. Fix it, and never report this as a bug. |99 | `unsat` | — | "None within len ≤ N", not "safe". |100 | `unknown` | — | Simplify the model. Never count it as `unsat`. |1011026. **Fix, and keep the harness.** Patch the code, update the model and expect103 `unsat`, and keep the reproducer as a regression test. Add the strict104 harness to the project's existing fuzz targets and property tests.105106## Verification107108Every trap you didn't strike ends as one of:109110- a reproducer that fails at that trap inside the strict harness111- `unsat` with a stated bound, plus a negative control that went `sat`112- unmodelled, with the reason113114For the negative control, remove one guard or widen one field and expect115`sat`. If the model stays `unsat`, it is over-constrained and its `unsat`116means nothing.117118## Failure modes119120- **A `sat` input doesn't fail as predicted.** The model's precondition is121 wrong. On 2026-11-13, a Python model treated every negative index as a122 silent wrap, and its input `b'\x00\xc0'` raised `IndexError` instead.123- **A later trap's `sat` input fails at an earlier line.** Use124 `in_order=True`. On 2026-11-13, the Go line-16 input panicked at line 14.125- **Every trap is `unsat`, including the buggy ones.** A guard was modelled126 with wide arithmetic where the source uses narrow. Run the negative control.127- **The strict harness passes, but Z3 says `sat`.** The wrapper covers only128 the object you wrapped. Buffers the function builds itself are unwrapped.129- **Random testing finds nothing, even when strict.** Check the random source.130 On 2026-11-13, a JS LCG that lost precision past 2^53 never hit the bug.131- **clang fails to link ASan or libFuzzer.** `libclang-rt-<N>-dev` is132 missing. Install it, or use gcc.