Benchmark

NumSharp Benchmark Harness

The exhaustive guide is benchmark/CLAUDE.md (architecture, every suite, config, troubleshooting). This skill is the distilled map + the actionable playbooks. Read benchmark/CLAUDE.md when you need depth beyond this.

THE convention: NPY/NS (memorize this)

ratio = NumPy_ms / NumSharp_ms. >1 = NumSharp faster, <1 = slower, =1 = parity. Higher is better. Timing basis: best window (min) — ratios compare each side's per-case minimum, never the mean. Interference (GC pauses, page-fault storms, ambient machine load) only ever adds time, and it lands almost entirely in the NumSharp side's right tail, so comparing means turned machine state into fake ratio regressions. Min-based harnesses run each case to a ~200 ms time budget (a >20 ms/call op runs exactly 100 times); the C# op-matrix keeps BenchmarkDotNet's 50 iterations. Per-case means stay in the JSON (numpy_mean_ms/numsharp_mean_ms) as tail diagnostics — a large mean/min gap on one side flags a contaminated window.

Used everywhere — matrices, geomeans, commit messages, every *_sheet.py. The report's status icons (✅/🟡/🟠/🔴 faster→slower · ▫ negligible/excluded · ⚪ pending · ❌ crashed) and their exact cutoffs live in references/run-and-report.md; the source of truth is get_status() + classify() in scripts/merge-results.py, so read those rather than trusting a hardcoded threshold. (The legacy run-benchmarks.ps1 prints the INVERSE NS/NPY — prefer NPY/NS for anything new.)

THE pitfall: Debug taints timings ~2×

Ad-hoc dotnet run file.cs / dotnet_run (file-based apps) compile both the script AND any #:project NumSharp.Core in Debug (DebuggableAttribute(DisableOptimizations)), which the JIT honors even over [MethodImpl(AggressiveOptimization)]. Hand-written C# hot loops run ~2× slow; IL-emitted kernels look normal. Every timing script MUST run dotnet run -c Release - < script.cs. #:property Optimize=true fixes only the script assembly, not Core. The BenchmarkDotNet projects are exempt (they mandate -c Release).

Structure — two sides, joined on (op, dtype, N)

run_benchmark.py --depth pass|light|measure is the execution-depth contract shared by BDN and NumPy (defined once in scripts/benchmark_modes.py, so the two languages can't drift). pass is an execution gate (one workload call, zero warmups); light uses 8 BDN measurements after 3 warmups plus one-sixth of NumPy's normal time budget; measure is the full 50-measurement / 5-warmup BDN + full NumPy publication profile. Pass/light write only under benchmark/results/ and must never replace root/docs/history artifacts. --dtypes f32,float64 selects comma-separated dtypes/aliases (canonical set + aliases in benchmark_modes.py); a multi-dtype scenario (for example src→dst casts) matches when any dtype side is requested.

The join is by normalized op name: normalize_op_name strips the dtype tag, [annotations], and identifier-only arg lists — so C# "np.foo(a)" and NumPy "np.foo" both collapse to np.foo and join. Get the names to normalize identically or the row shows as "C# not run" / "NumPy only".

The matrix + subsystems

• Op matrix — 18 comparison suites, each a C# namespace filter in run_benchmark.py's SUITES map (arithmetic, unary, reduction, broadcast, creation, manipulation, slicing, comparison, bitwise, logic, statistics, sorting, linalg, selection, fft, random, ndarray, api). Universal-tier rule: an operation/dtype with both 1K and 100K must also schedule 10M; the official merge checks NumPy and C# independently. Scalar/1K-only dispatch cases are the only intentional non-throughput exception. Allocation-heavy families map the 10M label to 1M physical elements symmetrically on both sides; 1M is not a separate report/dashboard tier.
• Backend profiles — the complete official LinearAlgebra BDN classes run under both the Core-only executable and NumSharp.Benchmark.CSharp.OpenBLAS; a merge gate rejects any missing exact-cell OpenBLAS peer. benchmark/backends/backend_profiles.py supplements backend-only product/LAPACK routes with the same schema; MissingBackendException and NotSupportedException are availability outcomes. Every profile publishes 1K / 100K / 10M; operation-specific physical work remains bounded (LAPACK maps those tiers to matrix sides 32 / 96 / 128). Separate profile JSON files are merged into one effective dataset, with actual_backend: managed retained for controls that never dispatch to BLAS.
• Five appended subsystems (own result models, appended not merged): nditer (iterator machinery), layout (op × 8 memory layouts × dtype), operand (1-D/scalar/mixed/broadcast), cast (astype 15×15 × layout), and fusion (np.evaluate). Each is a *_bench.{cs,py} pair + a *_sheet.py renderer.

Playbook — add a benchmark for a new op

The most common task. Full worked example in references/add-benchmark.md. In brief:

1. C# side — add a [Benchmark(Description = "np.foo(a)")] method to the class in Benchmarks/<Category>/ that fits (or a new class in that namespace so the suite's *Benchmarks.<Category>.* filter auto-includes it). Use BenchmarkBase (single-dtype, float64) like the manipulation classes, or TypedBenchmarkBase (dtype-swept) like arithmetic. [Params(Medium, Large)] for size.
2. NumPy twin — append to the matching run_<suite>_benchmarks(...) in numpy_benchmark.py, setting r.name, r.category, r.suite, r.dtype. Make .name normalize to the C# Description ("np.foo" ↔ "np.foo(a)").
3. Smoke it (this is usually the right scope — a full measured run is the post-release CI job): python benchmark/run_benchmark.py --depth pass --suites <suite> --dtypes <dtype> executes each selected BDN/NumPy cell exactly once and fails on workload errors. Use --depth light for a rough ratio.
4. Full numbers come from python run_benchmark.py --depth measure (or the benchmark.yml post-release workflow).

Other tasks → where to go

• Run the suite (official / subset), interpret the report, the reports/UI surfaces + snapshots → references/run-and-report.md. (The human-facing UI is the DocFX page docs/website-src/docs/benchmarks-dashboard.md; its Function Explorer data is generated, while narrative cards are curated. Generated dashboard data lives on the orphan data branch and is consumed at docs-build via the refs/data submodule (floated to the branch tip); publisher in tools/dashboard_data/ — see references/run-and-report.md.)
• Add or edit a matrix subsystem or backend profile case → references/subsystems.md.
• "Is np.<foo> benchmarked yet? What's still missing?" → the generated coverage ledger benchmark/coverage/generated/summary.md (+ coverage.{json,csv}), refreshed by scripts/audit_coverage.py from the [Benchmark(Description)] attributes across the op-matrix namespaces plus the reviewed coverage/overrides.json. Its "Missing benchmark coverage" table is the to-do list; its route map says which APIs need OpenBLAS. This is a source-level audit, not a timing run.
• Everything else (all suites, config internals, troubleshooting, type map) → benchmark/CLAUDE.md.

Gotchas

• BenchmarkDotNet's out-of-process toolchain fails here ("project names need to be unique") because sibling .claude/worktrees/ checkouts hold same-named benchmark projects. The official run uses InProcessEmit (OfficialBenchmarkConfig). For a smoke check use --list flat (reflection only, no toolchain) rather than trying to run BDN ad-hoc.

• What we commit is benchmark/history/<date>_<sha>/, not the gitignored benchmark/results/<ts>/ scratch. Reference benchmark/history/latest/benchmark-report.md.

• A row shows "C# not run" / "NumPy only"? The two sides' names didn't normalize to the same join key — the first thing to check for any new/renamed benchmark. scripts/check_smoke_joins.py verifies every C# [Benchmark(Description)] ↔ NumPy name join in both directions using the exact merge normalizer (from a quick numpy_benchmark.py --quick --size small smoke run), so it catches the mismatch without a full measured run.

• These are mostly view ops → sub-µs. flip/rot90/transpose-aliases are O(1) views; their benchmark tracks allocation/dispatch overhead, not throughput. benchmark/scripts/credibility.py marks every reviewed O(1)-in-N scenario negligible regardless of measured duration, and every dashboard rollup must honor that status. The proof ledger is benchmark/O1_EXCLUSIONS.md. Ops doing real work (trim_zeros, reductions) are where ratios are meaningful.

• A 30–50 % faster excursion in a few BDN iterations that never reproduces is turbo boost, not code. The host is a hybrid i9-13900K whose High-performance plan boosts Aggressively (~1.7× clock swing, hidden from powercfg -query until unhidden); BDN keeps fast outliers, so the lucky window becomes Statistics.Min. run_benchmark.py now locks the clock (boost Disabled, restored on exit) and pins BOTH languages to one P-core (NUMSHARP_BENCHMARK_AFFINITY, default 0x4) — see benchmark/scripts/benchmark_host.py; --no-lock-clock / --no-pin-core opt out — and it covers EVERY runner (sheets, backend profiles and the nditer harness pin themselves after their build, and every dotnet run - script / Python twin inherits the mask at spawn). Both languages also trim outliers on BOTH tails (OutlierMode.RemoveAll in every BDN config, Tukey k=1.5 in numpy_benchmark.py). Pinning trap: a bare ctypes.windll.kernel32.SetProcessAffinityMask(GetCurrentProcess(), …) (no argtypes/restype) fails silently — always read the mask back.

References

• references/add-benchmark.md — the detailed add-a-benchmark playbook (C# + NumPy twin + join-key rules + smoke).
• references/run-and-report.md — running the official run / subsets, the report + history snapshots, InProcessEmit.
• references/subsystems.md — backend profiles plus the five appended subsystems.