Performance Testing

Performance Testing

Overview

The performance-tests/ package measures Handsontable rendering and interaction performance using Playwright + Chrome DevTools Protocol (CDP) traces. It is a standalone package outside the pnpm workspace -- it has its own package.json, node_modules, and ESLint config.

Each scenario traces a specific user interaction (scrolling, filtering, sorting, editing), parses the CDP trace into DevTools-equivalent categories (scripting, rendering, painting, idle), and produces a compact markdown PR comment plus a self-contained interactive HTML report comparing against a golden baseline from develop.

Package Structure

performance-tests/
  scripts/run.mjs           # Orchestrator: build HOT UMD -> copy to fixtures -> run Playwright
  scripts/replay-goldens.mjs # Replays gh-pages develop goldens to re-derive the callout thresholds
  playwright.config.ts       # Sequential, 1 worker, 5 min timeout, chromium only
  trace-parser.mjs           # CDP trace -> DevTools-equivalent category breakdown
  .eslintrc.js               # Extends root config, relaxes JSDoc/console/await-in-loop rules
  lib/
    setup.mjs                 # Playwright globalSetup: records Chromium build + machine to output/environment.json
    environment.mjs           # Run provenance and the baseline compatibility key (Chromium, harness version)
    trace-runner.mjs          # CDP Tracing.start/stop + warmup/iteration loop + progress output; owns HARNESS_VERSION
    hook-timing.mjs           # Hook pair timing (inject/get/save) for before/after measurements
    snapshot-store.mjs        # Golden baseline save/load; refuses an incompatible baseline and says why
    median-snapshot.mjs       # Synthesizes the rolling-median develop baseline (+ its run-to-run spread)
    thresholds.mjs            # Shared classification logic (regression/improvement thresholds)
    chart-generator.mjs       # Inline SVG horizontal bar charts (base64 data URIs)
    report-builder.mjs        # Compact markdown PR comment (summary table + regression callouts)
    html-report-builder.mjs   # Self-contained interactive HTML report (inline CSS + JS)
    build-history-index.mjs   # Generates gh-pages history listing for develop runs
    teardown.mjs              # Playwright globalTeardown: parse traces -> report
    fs-utils.mjs              # Shared filesystem helpers (exists)
    scroll-utils.mjs          # Scroll-and-wait helpers (scrollToRow, scrollToColumn)
  scenarios/
    <name>/
      scenario.config.mjs     # { name, warmupRuns, iterations }
      fixture.html             # Standalone HTML loading HOT UMD from ../../fixtures/
      <name>.spec.ts           # Playwright test using runTracedScenario()
  fixtures/
    .gitkeep                   # Built JS/CSS copied here by run.mjs (gitignored)
  golden/                      # Golden snapshots (gitignored, fetched from gh-pages branch)
  output/                      # Trace JSONs + result.md + report.html (gitignored)

Adding a New Scenario

Every scenario needs exactly three files in scenarios/<name>/:

1. scenario.config.mjs

// Grid: <rows> x <cols> -- <rationale for this grid size>
export default {
  name: 'my-scenario',
  warmupRuns: 1,
  iterations: 3,
  // Bump when this spec changes what the marked window contains, or when `iterations` changes: the
  // median baseline only draws on develop goldens recorded at the same version (see
  // lib/environment.mjs).
  measurementVersion: 1,
};

measurementVersion covers everything scenario-local that changes the published number or its spread: the window's contents, and the iteration count (a mean of five and a mean of three have the same expected value but different spreads, and CV run would mean something different per row). HARNESS_VERSION covers runner-wide changes. When both change in one PR, the harness bump already restarts the whole golden pool and the scenario bump is not needed on top; say so in the config comment.

The name must match the directory name -- it determines the output/<name>/ subdirectory, and the teardown reads measurementVersion from the config by that name. Add a comment documenting the grid size and why it was chosen.

2. fixture.html

A standalone HTML file that creates a Handsontable instance and exposes it as window.__hot:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>My Scenario</title>
  <link rel="stylesheet" href="../../fixtures/handsontable.css">
  <script src="../../fixtures/handsontable.full.js"></script>
</head>
<body>
  <div id="hot"></div>
  <script>
    const hot = new Handsontable(document.getElementById('hot'), {
      data: Handsontable.helper.createSpreadsheetData(5000, 10),
      rowHeaders: true,
      colHeaders: true,
      width: 1280,
      height: 600,
      autoRowSize: false,
      autoColumnSize: false,
      licenseKey: 'non-commercial-and-evaluation',
      // Add scenario-specific options (filters, columnSorting, etc.)
    });
    window.__hot = hot;
  </script>
</body>
</html>

Important rules for fixtures:

• Always set autoRowSize: false and autoColumnSize: false -- these async plugins interfere with measurements.
• CSS path is ../../fixtures/handsontable.css (not handsontable.full.css).
• JS path is ../../fixtures/handsontable.full.js.
• Always expose the instance as window.__hot.

3. Spec file (<name>.spec.ts)

import { test } from '@playwright/test';
import path from 'node:path';
import { runTracedScenario } from '../../lib/trace-runner.mjs';
import config from './scenario.config.mjs';

const fixturePath = path.resolve(import.meta.dirname, 'fixture.html');

test(config.name, async({ page }) => {
  await page.goto(`file://${fixturePath}`);
  await page.waitForFunction(() => (window as any).__hot, undefined, { polling: 100 });

  // Optional pre-trace setup (e.g., scroll to position)

  await runTracedScenario({
    page,
    warmupRuns: config.warmupRuns,
    iterations: config.iterations,
    outputDir: path.resolve('output', config.name),
    actionFn: async() => {
      // The measured action goes here
    },
    // Optional: resetFn to restore state between iterations
  });
});

The runTracedScenario function handles:

• Creating the output directory
• Running warmup iterations (no tracing) with progress output
• Running measured iterations with CDP tracing and heartbeat dots
• Writing trace JSON files as iteration-{n}.json

Adding hook timing (filtering/sorting scenarios)

For scenarios that measure a specific hook pair (e.g., beforeFilter -> afterFilter):

import { injectHookTimer, getHookTiming, saveHookTimings } from '../../lib/hook-timing.mjs';

// Before tracing, inject the timer:
await injectHookTimer(page, 'beforeFilter', 'afterFilter');

const hookDeltas: number[] = [];
const outputDir = path.resolve('output', config.name);

// In afterActionFn -- NOT actionFn. getHookTiming is a page.evaluate, and inside
// actionFn its CDP round trip falls inside the marked window, so it is measured as
// part of the operation on measured iterations only:
afterActionFn: async() => {
  const timing = await getHookTiming(page, 'beforeFilter', 'afterFilter');
  if (timing.deltaMs != null) {
    hookDeltas.push(timing.deltaMs);
  }
},

// Inside resetFn, call injectHookTimer again to reset the store
// (it is idempotent -- prevents duplicate listener registration):
await injectHookTimer(page, 'beforeFilter', 'afterFilter');

// After runTracedScenario, save hook timing:
await saveHookTimings(outputDir, hookDeltas);

The saveHookTimings helper computes the average and writes hook-timing.json. The teardown automatically picks it up from each scenario directory.

Using scroll helpers

For scenarios that need to pre-scroll the grid (e.g., scroll-up starts from the bottom):

import { scrollToRow, scrollToColumn } from '../../lib/scroll-utils.mjs';

// Scroll to row 4999 and wait until it's rendered (no arbitrary timeouts)
await scrollToRow(page, 4999);

// Or scroll to column 4999
await scrollToColumn(page, 4999);

These combine scrollViewportTo() with a deterministic waitForFunction that checks the index mapper -- always use them instead of waitForTimeout.

Existing Scenarios

Iterations: 3 for the scroll and cell-editing scenarios, 5 for filtering, sorting, initial-load and source-data-validator-load. Each of those four states its own reason in its scenario.config.mjs (short windows on a 300 to 350 MB heap where one GC pause moves a mean of three by 10 to 20%; for source-data-validator-load, a 20% run-to-run spread after removing the runner factor), and their iterations are cheap next to the fixture load. lib/__tests__/scenario-configs.test.mjs pins the counts, so change the config and the test together. Do not raise the scroll scenarios: each iteration is 500 wheel round trips.

Run Commands

# Full pipeline (build HOT + copy fixtures + run all scenarios)
cd performance-tests && node scripts/run.mjs

# Run specific scenario only (fixtures must exist)
npx playwright test --grep "scroll-down"

# Golden mode (saves baseline snapshot)
PERF_MODE=golden node scripts/run.mjs

# Compare mode (loads golden, generates delta report)
PERF_MODE=compare node scripts/run.mjs

# Lint
npm run lint

# Type-check spec files
npm run typecheck

Build artifacts (handsontable/dist/ and handsontable/styles/) must exist before running tests directly with npx playwright test. The scripts/run.mjs orchestrator handles building automatically.

CI Workflow

The GitHub Actions workflow (.github/workflows/performance-tests.yml) operates in two modes:

• push to develop: Runs all scenarios in golden mode, deploys snapshots.json + report.html to the gh-pages branch under performance-reports/develop/<timestamp>/. Updates latest.json as a pointer for PR comparisons. Builds a history index page listing all past runs with their commit, Chromium build and CPU. The run is also compared against the trailing median of compatible develop goldens, for its report, the job summary and one ::warning annotation per regressed scenario -- so a shift on develop is seen on the develop run that introduced it. The snapshot it saves is never derived from history (teardown saves before it loads).
• pull_request: Fetches the last 20 develop goldens from gh-pages into golden/history/ (plus latest.json as a single-file fallback), runs all scenarios in compare mode, posts a compact sticky PR comment with a summary table and regression callouts, and deploys the full HTML report to performance-reports/<branch-slug>/ on GitHub Pages.

Push retries with rebase (up to 3 attempts) protect against concurrent gh-pages writes.

Baseline compatibility and provenance

Replaying the develop goldens showed that the numbers move for reasons unrelated to the grid: a Playwright bump (new Chromium) shifted initial-load by -18% and sorting by -20% in one develop push, and the median-of-5 baseline carried the old browser's numbers for five more pushes, so unrelated PRs were told they had regressed. Three mechanisms now address that, all in lib/environment.mjs:

• Provenance. lib/setup.mjs (Playwright globalSetup) records the Chromium build (browser.version()), CPU model and count, memory, platform and the GitHub runner image into output/environment.json. The teardown stamps it on the snapshot (environment, harnessVersion) and both reports print it in their footer.
• Compatibility key. { chromium, harnessVersion } at the snapshot level, measurementVersion per scenario. computeMedianSnapshot(..., { compatibleWith }) draws only on goldens with the same key, and within a scenario only on entries at the same measurementVersion. A golden recorded before provenance existed has no key and is excluded by the absence of the fields, like a pre-marks golden is by windowSource. With no compatible golden at all the comment says "no comparable baseline" and why (describeKeyMismatch, or one of BASELINE_REFUSALS in snapshot-store.mjs: empty history, no marks-valid golden, every scenario redefined, disjoint scenarios, an empty latest.json). With exactly one, the single-file fallback serves and the footer says "single develop run"; the median needs two. So deltas resume with the next develop push, and as a median once two have run. Bump HARNESS_VERSION in trace-runner.mjs when the runner changes what a window contains (settle, GC between iterations, work moved in or out); bump a scenario's measurementVersion when only that spec's definition changes.
• Run shift. The comment's Δ vs shift column and the footer's Run shift line name the median delta across scenarios: how much faster or slower this runner ran than the baseline's, which every scenario shares (the per-run factor spans 0.63x-1.12x across develop goldens and removing it takes the scroll scenarios' run-to-run CV from ~13% to ~3%). It is reported, never gated on: the callouts fire on the raw Δ Total, because a change that slows every scenario alike is exactly what a median cannot see.

scripts/replay-goldens.mjs groups goldens by key before replaying, so a replay across a Chromium change never medians two environments together; its Compatibility groups block shows how many goldens sit in each.

The Trace Pipeline

Understanding the data flow helps when debugging or extending:

The measured window (read before adding a scenario)

Everything published describes the slice between the two performance.marks that runTracedScenario() writes around the action. Two rules follow:

• No harness round trips inside actionFn. A page.evaluate that reads a value back is measured as part of the operation. Put readbacks in afterActionFn, which runs after the end mark.
• A resetFn must leave no frame behind. The runner settles after setupFn and resetFn for this reason, and skipSettle does not turn those off. Note that scrollToRow/scrollToColumn report trimming, not scroll position, so their waitForFunction returns before the scroll has rendered.

A category measured as exactly 0, or a CV of sqrt(n) × 100% (one nonzero iteration among zeros, with the sample standard deviation calcCv uses: 173.21% at three iterations, 223.61% at five), means the window is wrong -- not that the operation was cheap.

1. Spec calls runTracedScenario() -> forced GC (over a control CDP session), CDP Tracing.start, start mark, action, settle, end mark, afterActionFn, forced GC + Runtime.getHeapUsage readback, Tracing.end -> raw JSON per iteration, plus heap-after-gc.json for the scenario. The GC before tracing keeps the previous reset's garbage out of the window (initial-load's third iteration read ~50% slower than its first two before it); the readback after the end mark is the live set, recorded as updateCounters.jsHeapAfterGcBytes. Both are part of HARNESS_VERSION 2.
2. Teardown (lib/teardown.mjs) discovers output/*/iteration-*.json, calls parseTrace() from trace-parser.mjs
3. trace-parser.mjs categorizes events into DevTools categories (scripting, rendering, painting, loading, system, idle), measures the window runTracedScenario() marked around the action (falling back to the auto-zoomed window only for traces recorded without marks), synthesizes ProfileCall scripting from CPU profile data
4. Teardown averages across iterations via averageParsedTraces(), collects per-iteration values for CV% calculation, strips internal fields (_iterationValues, _debug) from saved snapshots
5. report-builder.mjs assembles a compact markdown PR comment; html-report-builder.mjs generates a full interactive HTML report with inline SVG charts from chart-generator.mjs
6. If PERF_MODE=golden: snapshot-store.mjs saves averaged results, deployed to gh-pages
7. If PERF_MODE=compare: teardown loads golden from gh-pages, report shows deltas

Shared Utilities

The lib/ directory provides reusable helpers to avoid duplication across scenarios:

Always import from these shared modules rather than duplicating logic in scenario specs.

.mjs Convention

All .mjs files in this package follow the node-scripts-dev skill conventions:

• node: prefix for builtins
• node:fs/promises async APIs (never sync)
• import.meta.dirname for paths
• join() from node:path for cross-platform paths
• No TypeScript syntax -- .mjs files are plain JavaScript. Use /** @type {any} */ JSDoc casts, not as any.

Common Mistakes