Benchmark protocol
Benchmarks serve two purposes: regression gates (internal, every release) and published comparisons (external, credibility). Both die if the methodology is sloppy — a benchmark we can't defend in a hostile Hacker News thread must not be published.
Metrics that matter (for a compiler)
| Metric |
How |
| Compile time |
per-component and full-catalog wall clock, cold + incremental (compileIncremental reuse) |
| Emitted output size |
bytes raw + gzip per target, per component and for the full catalog CSS/JS |
| Output overhead |
emitted component vs a hand-written idiomatic equivalent: LOC, bundle delta, render performance |
| Runtime cost |
zero-runtime promise: no @inkline/core in output; wrapper cost where one exists (__InkTransition) |
| Source-map cost |
compile-time and size delta of sourceMap: external/inline/none |
| Emitted-code type feedback |
time-to-typecheck of the generated per-target output (conformance path) |
Existing surface (extend, don't duplicate)
core/compiler/scripts/bench.ts — tinybench suite run via pnpm bench (in core/compiler), with baseline comparison through the runBenchSuite harness (@inkline/compiler/testing). This is the number source for perf claims in PRs. Output-size checks ride the build artifacts (ui/*/.inkline/, framework dist/); pixel-behavior regression rides the e2e visual-parity suite (see adversarial-qa).
Fairness rules (non-negotiable for published numbers)
- Same reference component set per contender, idiomatic for each (no strawmen) — equivalent props, states, slots, styling.
- Pinned versions, stated hardware, median of ≥5 runs after warmup; report variance.
- Measure the path users actually run (CLI compile for libraries, plugin transform for apps) and say which.
- Contenders: Mitosis (the direct write-once-compile-everywhere neighbor) and hand-written per-framework components (the honest baseline every team compares against). Web-component wrappers (Lit/Stencil) only for "one codebase" claims, clearly framed — they ship a runtime, we don't.
- Publish the harness with the numbers. Reproducibility is the argument.
- Report losses honestly. A benchmark Inkline never loses reads as marketing and converts nobody. "Hand-written is N% smaller on target X" is a finding, not a failure — it's the compiler's backlog.
Regression gating
Per release train: run pnpm bench against the previous tag's baseline; track full-catalog compile time and per-target output size. Deltas beyond noise (>5% compile time, >2% output size) → flag on the release issue; Keeper holds the train until explained (regression vs accepted trade documented in the changeset). Any PR claiming "faster"/"smaller" carries its numbers, and @gauntlet re-measures independently before the claim stands.
Reporting format
Environment: <hw, os, node, pnpm, versions table>
Scenario: <component set, targets, cold/incremental, path (CLI|plugin)>
Results: <table, median ± spread>
Interpretation: <one paragraph, includes where we lose and why>
Harness: <repo/branch link, exact commands>
1---2name: benchmark-protocol3description: How Inkline measures itself — compile performance, output size and quality vs hand-written components and Mitosis, fairness rules, and reporting format. Use for perf work, regression gates, and published comparisons.4---56# Benchmark protocol78Benchmarks serve two purposes: **regression gates** (internal, every release) and **published comparisons** (external, credibility). Both die if the methodology is sloppy — a benchmark we can't defend in a hostile Hacker News thread must not be published.910## Metrics that matter (for a compiler)1112| Metric | How |13| -------------------------- | ----------------------------------------------------------------------------------------------------- |14| Compile time | per-component and full-catalog wall clock, cold + incremental (`compileIncremental` reuse) |15| Emitted output size | bytes raw + gzip per target, per component and for the full catalog CSS/JS |16| Output overhead | emitted component vs a hand-written idiomatic equivalent: LOC, bundle delta, render performance |17| Runtime cost | zero-runtime promise: no `@inkline/core` in output; wrapper cost where one exists (`__InkTransition`) |18| Source-map cost | compile-time and size delta of `sourceMap: external/inline/none` |19| Emitted-code type feedback | time-to-typecheck of the generated per-target output (conformance path) |2021## Existing surface (extend, don't duplicate)2223`core/compiler/scripts/bench.ts` — tinybench suite run via `pnpm bench` (in `core/compiler`), with baseline comparison through the `runBenchSuite` harness (`@inkline/compiler/testing`). This is the number source for perf claims in PRs. Output-size checks ride the build artifacts (`ui/*/.inkline/`, framework `dist/`); pixel-behavior regression rides the e2e visual-parity suite (see `adversarial-qa`).2425## Fairness rules (non-negotiable for published numbers)26271. Same reference component set per contender, idiomatic for each (no strawmen) — equivalent props, states, slots, styling.282. Pinned versions, stated hardware, median of ≥5 runs after warmup; report variance.293. Measure the path users actually run (CLI compile for libraries, plugin transform for apps) and say which.304. Contenders: **Mitosis** (the direct write-once-compile-everywhere neighbor) and **hand-written per-framework components** (the honest baseline every team compares against). Web-component wrappers (Lit/Stencil) only for "one codebase" claims, clearly framed — they ship a runtime, we don't.315. Publish the harness with the numbers. Reproducibility is the argument.326. Report losses honestly. A benchmark Inkline never loses reads as marketing and converts nobody. "Hand-written is N% smaller on target X" is a finding, not a failure — it's the compiler's backlog.3334## Regression gating3536Per release train: run `pnpm bench` against the previous tag's baseline; track full-catalog compile time and per-target output size. Deltas beyond noise (**>5% compile time, >2% output size**) → flag on the release issue; Keeper holds the train until explained (regression vs accepted trade documented in the changeset). Any PR claiming "faster"/"smaller" carries its numbers, and @gauntlet re-measures independently before the claim stands.3738## Reporting format3940```41Environment: <hw, os, node, pnpm, versions table>42Scenario: <component set, targets, cold/incremental, path (CLI|plugin)>43Results: <table, median ± spread>44Interpretation: <one paragraph, includes where we lose and why>45Harness: <repo/branch link, exact commands>46```
inkline