nubjs benchmarking
Honest, reproducible comparative install benchmarks of nub install against npm / pnpm / bun — the external, wall-clock-and-file-count method. For decomposing where the time goes INSIDE a single nub install, use pm-perf-tracing instead.
Applies to: benchmarking against another PM, refreshing homepage/blog install numbers, verifying a perf claim before it ships. A single non-genuine cell discredits the whole table.
The tool: hyperfine
/opt/homebrew/bin/hyperfine is the canonical timer. The load-bearing flag is --prepare, which runs setup BEFORE each timed run, untimed. Never time setup.
# COLD: empty the tool's REAL cache + wipe node_modules before each run (untimed), then time the install.
hyperfine --warmup 0 --runs 5 \
--prepare 'rm -rf node_modules && d="$(bun pm cache)" && rm -rf "${d:?}"' \
'bun install --ignore-scripts'
# WARM-RELINK: cache populated, wipe ONLY node_modules before each run.
hyperfine --warmup 1 --runs 5 \
--prepare 'rm -rf node_modules' \
'nub install --ignore-scripts'
# WARM-SAT: node_modules already present (idempotency path) — no prepare wipe.
hyperfine --warmup 1 --runs 5 'nub install --ignore-scripts'
Prefer median + spread (min–max) over the mean — contention skews the mean.
The cold / warm protocol
node_modules is deleted before EVERY timed run, cold and warm, always in --prepare. The cold/warm axis differs only in global-cache state:
- cold — the tool's real cache is EMPTY (genuine download).
- warm-relink — cache populated,
node_moduleswiped (the link-from-store path; the homepage number). - warm-sat —
node_modulesalready present; a separate, clearly-labeled scenario, not the headline.
GENUINE-COLD per tool — the cache is on DISK
A "cold" run is only cold if the tool's real on-disk cache is gone. Setting a cache-dir env var does not guarantee that — bun ignores BUN_INSTALL_CACHE_DIR and $HOME, resolving its cache via the OS passwd home. You must wipe the disk path.
| tool | real cache path | clear command for cold |
|---|---|---|
| nub | its store (NUB_CACHE_DIR + XDG_DATA_HOME/XDG_CACHE_HOME) |
rm -rf "$NUB_CACHE_DIR" "$XDG_DATA_HOME" "$XDG_CACHE_HOME" |
| npm | ~/.npm/_cacache (or --cache <dir>) |
rm -rf <cache> (the --cache dir you pass) |
| pnpm | pnpm store path (or --store-dir <dir>) |
rm -rf <store> (or pnpm store prune) |
| bun | bun pm cache = the real ~/.bun/install/cache |
rm -rf "$(bun pm cache)" — env vars do NOT relocate it |
VERIFY each cold is genuine — the offline-fails check
After wiping a cache, an --offline install MUST FAIL. A pass means the cache wasn't cleared (wrong disk path), and any "cold" number from it is a warm-link artifact. Run this for every tool before trusting a cold number:
| tool | expected after cache wipe | genuine? |
|---|---|---|
| nub | rc≠0, "not available in the local cache" |
FAIL → genuine |
| pnpm | rc=1, ERR_PNPM_NO_OFFLINE_TARBALL |
FAIL → genuine |
| npm | rc≠0, cache-miss error |
FAIL → genuine |
| bun | if bun install --offline SUCCEEDS → cache NOT cleared → bun cold is NOT genuine |
A true bun cold needs a clean container, or wiping the user's real cache (destructive — don't, unless in Docker).
Apples-to-apples isolation
- Each tool gets its OWN cache dir.
--ignore-scriptsfor ALL tools (or--allow-scriptsfor all — same on both sides).- Identical fixture, identical lockfile throughout (use a
--frozen-lockfile/--frozenequivalent where the tool offers one). - Interleave tool order round-robin (nub → npm → bun → pnpm, repeat) — never all-of-one-then-the-other — so drift in host load hits every tool equally.
Load discipline — measure only on a quiet machine
- Check the load average before measuring and only proceed when the machine is quiet. A dedicated quiet box or a CI runner near zero load is ideal for anything that will be published.
- On a shared dev host, WAIT for load to fall below a threshold — don't measure above it. A practical gate is a 1-minute load average under ~40 (pick a threshold the machine actually reaches; below ~5 is ideal on a dedicated box). Poll, wait, then run; abort and retry if it spikes mid-run.
- Report the load that held during the run alongside the numbers.
- File counts and store-entry counts are exact and load-independent — a robust cross-check, and the clearest way to tell the dedup story. They complement a properly-measured time, not replace it.
File-count forensics (the load-independent crux)
find node_modules -type f -o -type l | wc -l # total materialized entries
ls -d node_modules/**/core-js node_modules/core-js # physical copies of a duplicated dep (dedup story)
Reach for this first — exact, reproducible, immune to load.
The honesty bar (anti-juicing)
- VERIFY every cold is genuine (offline-fails check) BEFORE citing it.
- NEVER compare one tool's genuine-cold to another's warm-link — the exact misleading comparison the offline check exists to prevent.
- Report what was actually measured, caveats included (which colds are genuine, host load, sample size).
- The homepage cites the WARM number because it is the honest, reproducible one.
- A single non-genuine cell discredits the whole table. When in doubt, exclude the cell and say why.
Process hygiene (this runs on the maintainer's machine)
Short-lived install/measurement processes reap themselves. The hazard is a long-lived process a bench starts and forgets:
- Never leave a dev server running. If a bench starts one, tear it down in the same run (
trap '…kill…' EXIT INT TERM). - Docker:
docker run --rm, and confirmdocker psis empty when done.
Reference template
/tmp/cs-bench-final.sh is a working 4-tool harness (nub / npm / bun / pnpm) — per-tool isolated cache dirs, --ignore-scripts for all, interleaved round-robin order, node_modules wiped before every run, median/spread helpers. Read its structure before hand-rolling a new one; adapt the cache paths and fixture, keep the protocol.
Internal decomposition
When a comparative number raises "WHY is nub's phase X slow?", switch to pm-perf-tracing: RUST_LOG=debug nub install for the phase:resolve/fetch/link split, and the gated AUBE_DIAG_FILE per-file linker strategy tally.