Oracle

NumSharp Differential-Fuzz Oracle

Mental model (why this exists)

NumPy 2.4.2 is the oracle. Python generates a committed, bytes-exact corpus of (inputs → NumPy output) cases; the C# harness rebuilds the exact operand bytes, runs NumSharp, and bit-compares against the recorded NumPy output. No Python runs at test time or in CI — the corpus is replayed. A green gate means NumSharp is byte-for-byte NumPy across every layout × dtype the corpus covers.

Three outcomes per case: bit-exact (pass), a documented divergence in MisalignedRegistry (excused, never silent), or a failure (red → real bug, auto-shrunk to a 1-element repro).

The authoritative narrative lives in the project .claude/CLAUDE.md → "Differential-Fuzz Pipeline (NumPy oracle)" and the divergence ledger test/NumSharp.Tests.Oracle/Fuzz/README.md. This skill is the actionable playbook.

File map

The op oracle lives in two directories: generators in test/oracle/, the replay harness + committed corpus in test/NumSharp.Tests.Oracle/Fuzz/. (Two sibling oracles — flags & layout-parity — live in the main test/NumSharp.Tests/ project instead; see "The six oracles" below.)

The gate

dotnet test --filter "TestCategory=FuzzMatrix" runs the gate classes:

• FuzzCorpusTests (+ .Kinds) — the op corpus: one [FuzzMatrix] method per tier. Includes deterministic creation/conversion/multioutput, value/parity tiers (specials/precision/products/fft/matmul_parity/ linalg_parity/poly/einsum/nan/nanscan/random_parity/generator_parity/...), and result-kind/error tiers (iter/dtype_text/out_where/errors_full, in .Kinds). This is where new-op work lands.
  • Two runners: RunCorpus(file) (strict everywhere) and RunHostLibmCorpus(file) — used by unary/nan/precision/fft/numpy_f32_kernels, hard-gated on Windows and Inconclusive off-Windows (their transcendental / FFT-twiddle / Vector<T> cells are win-amd64 CRT-libm and SIMD-width dependent). Every portable cell in those tiers is a deterministic NumSharp kernel, green on all platforms by construction.
  • FuzzRegression() enumerates Fuzz/corpus/regressions/*.jsonl at runtime and replays each. The subdir is created on demand (currently empty) — the nightly soak drops shrunk repros there to pin them forever.
• IndexOracleTests — advanced-indexing get/set (index_curated + index_dtype + index_setter_dtype + index_random).
• OracleSurfaceCoverageTests / OracleCoverageStrengthTests / Journey3TouchedOracleCoverageTests — coverage gates: every public np/np.linalg/np.fft/np.random method is classified; every ordinary op has ≥4 cases with ≥1 changing axis; all 186 journey3-touched callables have a direct case. A new unclassified API fails here.
• UndisposedIntermediateTests / NativeAllocationChokepointTests — oracle-free leak gates that reuse the corpus + OpRegistry to assert the buffer pool balances (zero-leak) and that no raw native-alloc site escapes the chokepoint allowlist. (ScopeAudit/[TestCategory("ScopeAudit")] is the shared pool-counter harness.)
• BlasBackendDeltaTests — replays only the ~1.7K BLAS-affected ordinary cases twice (managed vs OpenBLAS), deduplicates identical outcomes, byte-checks the real flips against NumPy on the pinned host.
• MetamorphicTests — NumPy-free invariants (round-trips / involutions), no oracle needed.
• HarnessSelfTests — proves the gate has teeth (BitDiff catches value/NaN/-0 diffs; the corpus is non-vacuous; the Shrinker reproduces a planted divergence). A green FuzzMatrix that skipped every case would fail here.
• OpRegistryRandomIsolationTests — asserts stateful random ops in the corpus don't mutate global np.random state.

Run one tier while iterating: dotnet test --no-build -f net10.0 --filter "FullyQualifiedName~FuzzCorpusTests.Manip".

Known flake: the full TestCategory=FuzzMatrix run may end with "Test host process crashed" (an intermittent AccessViolation) AFTER all tests report Passed. That's a teardown crash, not a failure — re-run the specific FuzzCorpusTests class (exit 0, no crash) to confirm green.

The six oracles

All share one philosophy (NumPy — or an independent scalar oracle — is truth; the corpus is committed; no Python at test time), but they are separate corpora + gates:

Oracles 5 & 6 live in the main test project (test/NumSharp.Tests/Backends/), not the Oracle project — they gate flags/view semantics rather than op values, and are NOT part of TestCategory=FuzzMatrix.

Playbook — add a new op to the oracle

This is the most common task. The full worked example (flip/trim_zeros, plus params/char/OpRegistry patterns) is in references/add-op.md — read it when adding an op. In brief:

1. Pick the tier in gen_oracle.py whose gen_<mode> fits your op (shape ops → gen_manip, elementwise → gen_binary/gen_unary, reductions → gen_reduce, …). Modes are listed in main()'s elif mode == ....
2. Add a job to that tier's job list: a (opname, params_dict, lambda v: np.<op>(v, ...)) tuple. Guard by nd/sz where NumPy would raise (the generator's try/except skips those and prints a count).
3. Add the matching case to OpRegistry.cs — case "<opname>": return np.<op>(ops[0], ...); — reading params with p["k"].GetInt32() / p["trim"].GetString() / ParseIntArray(p["axes"]). Convention: "axis" (scalar int) vs "axes" (int[]) selects the overload.
4. Regenerate the corpus (needs numpy==2.4.2): python test/oracle/gen_oracle.py <mode>.
5. Build (the csproj glob copies the corpus to test output) and run the tier: dotnet build then dotnet test --no-build -f net10.0 --filter "FullyQualifiedName~FuzzCorpusTests.<Tier>".
6. Triage any red (see below). Char coverage is woven automatically via char_tier(<mode>) — no extra wiring.

Other tasks → where to go

• Regenerate any/all tiers, or a dtype/layout question → references/regenerate.md (the full command matrix, the numpy pin, determinism, the ALL_DTYPES / char / decimal story, how layouts feed every op).
• A case diverged (red), or you need to excuse an intended difference → references/triage.md (bit-diff → shrink → MisalignedRegistry vs OpenBugs; NaN/Decimal comparison rules).
• Deeper system map (all generators, the corpus tiers, the six oracles, the leak/coverage gates, the harness classes) → references/architecture.md.

Critical gotchas (learned the hard way)

• Public-surface completeness is gated. OracleSurfaceCoverageTests reflects np/np.linalg/ np.fft/np.random and fails on a public method with no corpus key or explicit classification.
• Pin numpy==2.4.2. A different NumPy version can shift bytes and make the committed corpus wrong. Verify python -c "import numpy; print(numpy.__version__)" before regenerating.
• The corpus diff is huge but harmless. Case ids carry a global running counter ({op}/{layout}/{dtype}/{n}), so adding one job renumbers every following id. Expect a large git diff on *.jsonl — it's renumbering, not semantic churn.
• Char has no NumPy dtype. It rides the uint16 proxy: char_tier("<mode>") re-runs your gen_<mode> with the Char pool and relabels uint16 → char. Add your op to a gen_<mode> whose main() branch calls char_tier (18 of them — arith/divmod/comparison/unary×2/bitwise/reduce/scan/stat/manip/sort/tail/astype/where/logic/matmul/ rounding/copyto) and Char coverage is automatic. creation and conversion append their own proxy rows, bringing the committed total to 20 Char-bearing files; a mode with no proxy call (e.g. modf, place) has none.
• Decimal has no NumPy analog. It rides the independent C# oracle gen_decimal_oracle.cs (naive scalar System.Decimal), regenerated via dotnet run test/oracle/gen_decimal_oracle.cs. If your op needs Decimal coverage, add it there too.
• The generator resolves paths relative to test/oracle/ and writes into test/NumSharp.Tests.Oracle/Fuzz/corpus/. Run it from test/oracle/ (or with that CWD). CI replays the committed corpus and never runs the generator.
• OpRegistry's default: throws NotSupportedException(op) — so a corpus op with no registered case fails the tier loudly. If a tier goes red immediately on a new op, you forgot (or mistyped) the OpRegistry case.
• A new LAYOUT needs only the Python builder — there is NO LayoutCatalog.cs. FuzzCorpus.Reconstruct rebuilds any operand view from the serialized (dtype, shape, element-strides, offset, base-bytes) descriptor, so the C# side is layout-agnostic. Add the (base, view) builder to layout_catalog.py, regenerate the affected tiers — done. (The "mirror it in LayoutCatalog.cs" line in layout_catalog.py's own header is stale.)
• Non-array results have their own wiring. An op that returns a dtype, a string, a scalar, or a tuple is NOT registered in OpRegistry.Apply; it goes through OpRegistry.Kinds.cs (ApplyTuple/ApplyDtype/ApplyText) paired with gen_oracle.py's gen_dtype_text/gen_iter/gen_out_where, and the generator marks it with expected.kind (dtype/text/tuple/scalar) via _arr_expected(kind=…)/_tuple_expected. See references/architecture.md → "Result kinds".
• Registering an op also enters the ZERO-LEAK gate. UndisposedIntermediateTests replays the whole corpus through OpRegistry with every result disposed and asserts the buffer pool's takes==returns. The KnownEscapes registry is now empty — every op is gated at zero — so an op that strands a pooled intermediate (undisposed scratch, an orphaned AsGeneric, an operator leftover) turns this gate red even when its values are bit-exact. Fix the leak ([NDScoped]/dispose the intermediate), don't add a KnownEscapes entry. NativeAllocationChokepointTests separately fails if your op adds a raw NativeMemory.*/Marshal.AllocHGlobal site outside the chokepoint allowlist — route scratch through the pools.
• Some tiers go Inconclusive off-Windows, not red. unary/nan/precision/fft/numpy_f32_kernels (via RunHostLibmCorpus) and the host-pinned matmul_parity/random_parity_host/generator_parity_host record bytes reproducible only on win-amd64 (CRT libm, SIMD widths, the exact BLAS build). Off-Windows they assert Inconclusive, never a failure — expected, not a problem. Regenerate on your host to gate against it (see references/triage.md).

References

• references/add-op.md — the detailed add-an-op playbook with a worked example (params, char, OpRegistry patterns).
• references/regenerate.md — the full regeneration command matrix + dtype/layout coverage model.
• references/triage.md — divergence handling: bit-diff → shrink → MisalignedRegistry vs OpenBugs.
• references/architecture.md — the complete system map (generators, tiers, the six oracles, the leak/coverage gates, the harness).