Authoring Benchmarks

Authoring Benchmarks

Create browser benchmarks that answer a specific performance question. Browser behavior is the product behavior, so Chromium is the default runtime for component and utility benchmarks.

Required Context

Before changing benchmarks:

1. Read the testing overview.
2. Read the Bench section of the Vite internals documentation.
3. Read the target project's DEVELOPMENT.md.
4. Inspect the implementation, its unit tests, and the closest existing benchmark.

Use the guidance-build skill as well when adding or changing Wireit tasks. Use the audit-ci skill instead when measuring pnpm run ci, build orchestration, or the CI completion path.

Choose the Correct Performance Test

Do not use an SSR benchmark as a substitute for browser lifecycle performance. SSR serialization and browser DOM work are different workloads.

Benchmark Convention

• Name every benchmark file *.test.bench.ts.
• Run every benchmark through Chromium using libraryBenchConfig.
• Keep the benchmark beside the implementation it measures.
• Use one vitest.bench.ts configuration per project.
• Do not add .test.bench.browser.ts, a second browser configuration, or a Node-default benchmark convention.
• Use describe(Component.metadata.tag, ...) for component benchmarks and a descriptive subsystem name for utilities.
• Use action-and-scale labels such as filters 1,000 options to one match.

The shared configuration is @internals/vite/configs/bench.js. The Core reference configuration is projects/core/vitest.bench.ts.

Design the Workload First

State the performance question before writing benchmark syntax. A useful benchmark has:

1. A product-relevant operation: Filtering options, traversing a grid, synchronizing tree state, generating a path, or updating rendered icons.
2. A representative scale: Prefer enough data to expose algorithmic and allocation costs. A leaf rendering benchmark without scalable work does not justify its maintenance cost.
3. A stable input: Use fixed deterministic data. Avoid randomness, clocks, network requests, and machine-specific files.
4. Consistent iterations: Every sample must perform the same class of work. Alternate values to force updates without allowing state to grow indefinitely.
5. An observable completion point: Return computed values. For Lit updates, await elementIsStable() and any follow-up asynchronous update triggered by the component.
6. A narrow boundary: Exclude fixture creation, selectors, and data generation unless those operations are explicitly under test.

Use at least two sizes when the scaling behavior is the question. If comparing a single operation with a batch, normalize the batch mean per item before drawing conclusions.

Browser Fixture Pattern

Vitest suite hooks are not the benchmark lifecycle. Use BenchOptions.setup and teardown so fixtures exist during warmup and sampled runs.

import { html } from 'lit';
import type { BenchOptions } from 'vitest';
import { bench, describe } from 'vitest';
import { createFixture, elementIsStable, removeFixture } from '@internals/testing';
import { Combobox } from '@nvidia-elements/core/combobox';
import '@nvidia-elements/core/combobox/define.js';

const optionTemplates = Array.from(
  { length: 1_000 },
  (_, index) => html`<option value=${`${index % 2 ? 'odd' : 'even'}-${index}`}>Option ${index}</option>`
);

describe(Combobox.metadata.tag, () => {
  let element: Combobox;
  let fixture: HTMLElement;
  let input: HTMLInputElement;
  let searchIndex = 0;

  const options: BenchOptions = {
    throws: true,
    async setup() {
      fixture = await createFixture(html`
        <nve-combobox>
          <label>Benchmark</label>
          <input type="search" />
          <datalist>${optionTemplates}</datalist>
        </nve-combobox>
      `);
      element = fixture.querySelector<Combobox>(Combobox.metadata.tag)!;
      input = fixture.querySelector<HTMLInputElement>('input')!;
      await elementIsStable(element);
    },
    teardown() {
      removeFixture(fixture);
    }
  };

  bench(
    'filters 1,000 options',
    async () => {
      input.value = searchIndex++ % 2 ? 'even' : 'odd';
      input.dispatchEvent(new InputEvent('input', { bubbles: true }));
      await elementIsStable(element);
    },
    options
  );
});

Rules for every case:

• Set throws: true. Without it, a failing benchmark can report empty or NaN results instead of failing.
• Put fixture and data preparation in setup when they are outside the measured workload; setup time is not sampled.
• Perform synchronous cleanup in teardown with removeFixture().
• Use real registered custom elements and the real DOM. Mock only unavailable browser capabilities.
• Force actual work. Reassigning the current value can turn the benchmark into a no-op.
• Await completion. Measuring only the property assignment misses rendering and asynchronous synchronization.
• Keep cold-load and steady-state questions separate. Warmup and browser caching make benchmarks suitable for steady-state work; use Lighthouse for first-load behavior.

For a pure utility, keep deterministic input outside the callback and return the result:

bench('transforms 10,000 values', () => transformValues(values), { throws: true });

Avoid Invalid Measurements

Do not:

• Measure only Lit fixture overhead when the claimed target is component logic.
• Use beforeAll, afterAll, beforeEach, or afterEach to manage benchmark fixtures.
• Include assertions, console output, snapshots, or debug logging in the timed callback.
• Accumulate selected nodes, listeners, DOM children, or other state across samples.
• Compare hz directly between workloads that process different item counts.
• Treat one noisy local run as proof of a regression or optimization.
• Hide outliers by deleting samples or adding arbitrary delays.

If an operation is faster than the browser timer resolution, benchmark a fixed batch and normalize the result. A reported min of 0 is a timer-resolution warning, not zero-cost code.

Run Benchmarks

Run repository commands through mise. From the target project:

mise exec -- pnpm run test:bench

For one benchmark file, first ensure the project build is current, then run:

NODE_ENV=production mise exec -- pnpm exec vitest bench --run --config=vitest.bench.ts src/<feature>/<feature>.test.bench.ts

Do not run other browser-heavy tasks concurrently when collecting comparison data. For regression analysis, use the same machine, Chromium version, power state, command, inputs, and build mode. Collect at least three runs per revision and compare medians.

Interpret Results

Use mean or its inverse hz for central throughput, percentiles for tail behavior, and RME to judge confidence. Investigate high RME by checking machine contention, outliers, state accumulation, timer resolution, and inconsistent work. Rerun before changing implementation.

For a batch of N items:

per-item mean = batch mean / N
per-item throughput = N × batch hz

Report the command, browser/runtime context, workload size, median across runs, tail latency, RME, and any normalization. Describe observed changes as correlation unless one-variable experiments establish causality.

Add Benchmark Support to a Project

When a project has no benchmark task, follow the current Core setup rather than inventing another harness:

1. Add vitest.bench.ts using libraryBenchConfig and a source alias.
2. Add a Wireit-backed test:bench script with benchmark files and configuration as inputs and no outputs.
3. Depend on the builds required by the benchmark imports.
4. Exclude *.test.bench.ts from production build inputs and library TypeScript output.
5. Confirm unit-test discovery and coverage exclude benchmark files.
6. Document pnpm run test:bench in the project's DEVELOPMENT.md.

Benchmarking is opt-in unless the repository's CI policy explicitly adds it to a required workflow. Do not add unstable wall-clock thresholds without a baseline strategy and controlled runners.

Existing References

• Icon browser updates
• Sparkline transforms
• Combobox filtering
• Grid navigation and traversal
• Tree synchronization
• Select synchronization

Validation

After modifying benchmark code or configuration, run:

cd projects/<project>
mise exec -- pnpm run lint
mise exec -- pnpm run test
mise exec -- pnpm run build
mise exec -- pnpm run test:bench

Also run Prettier, Vale for changed Markdown, and git diff --check. Confirm no benchmark source appears in production dist/ output.