Adding Benchmarks

Adding Benchmarks

When to Use

Use this skill when:

• Adding new performance benchmarks to a package
• Creating benchmark tests that should be tracked over time
• Integrating existing benchmarks into the CI pipeline

Benchmark System Overview

Benchmarks flow through the system as follows:

1. Generation: Each package produces bench-out/*.bench.json files
2. Aggregation: bench_merge in root bootstrap.sh combines all files, prefixing names with the package path
3. Upload: CI caches the merged JSON and GitHub Action uploads to the benchmark dashboard
4. Display: Results appear at the dashboard with historical tracking

Live dashboard: https://aztecprotocol.github.io/benchmark-page-data/bench/?branch=next

How Benchmark Names Work

Name Construction

The final benchmark name combines two parts:

1. Package prefix (added automatically by bench_merge): Based on where the file lives
2. Local name (what you write in JSON): Your metric identifier

Dashboard Grouping

The dashboard splits names by / to create a collapsible tree. The last segment becomes the chart name, everything before it becomes the group hierarchy.

Naming Best Practices

Use / to create logical groupings:

[
  {"name": "Tx/private/getTxHash/avg", "value": 1.2, "unit": "ms"},
  {"name": "Tx/private/getTxHash/p50", "value": 1.1, "unit": "ms"},
  {"name": "Tx/public/getTxHash/avg", "value": 2.3, "unit": "ms"}
]

Avoid flat names - they create no hierarchy and are hard to navigate:

[
  {"name": "tx_private_gettxhash_avg", "value": 1.2, "unit": "ms"}
]

Common suffixes:

• Timing: avg, p50, p95, p99, min, max, total
• Size: _opcodes, _gates, memory
• Rate: gasPerSecond, jobs_per_sec

Required JSON Format

All benchmark files must be arrays using the customSmallerIsBetter format:

[
  {"name": "category/metric_name", "value": 12345, "unit": "gas"},
  {"name": "category/another_metric", "value": 100.5, "unit": "ms"}
]

Rules:

• Must be a JSON array [...], not an object
• Each entry needs name, value, unit
• value must be numeric (lower is better)
• File must end with .bench.json

Optional fields (preserved by benchmark-action):

• range (string): Variance info (e.g., "± 5%")
• extra (string): Metadata — used for stacked chart grouping (see below)

Stacked Charts

To render multiple metrics as a single stacked area chart (e.g., component breakdowns), add an extra field with a stacked:GROUP_NAME value. Entries sharing the same GROUP_NAME are overlaid on one chart.

[
  {"name": "proving/cpus-8/total_ms", "value": 31663, "unit": "ms"},
  {"name": "proving/cpus-8/oink_prove_ms", "value": 4992, "unit": "ms", "extra": "stacked:proving/cpus-8/components"},
  {"name": "proving/cpus-8/sumcheck_ms", "value": 3318, "unit": "ms", "extra": "stacked:proving/cpus-8/components"},
  {"name": "proving/cpus-8/circuit_ms", "value": 4642, "unit": "ms", "extra": "stacked:proving/cpus-8/components"}
]

How it works:

• extra: "stacked:GROUP_NAME" → entries with the same GROUP_NAME are rendered as one stacked chart
• No extra field → individual line chart (default behavior)
• Stacked entries still appear as individual charts on the main benchmark-action dashboard; the stacked view is rendered by a custom dashboard page
• The GROUP_NAME becomes the chart title (after bench_merge prefixing, same as name)
• The extra field is one of the 5 fields preserved by the benchmark-action Zod schema (name, value, unit, range, extra); any other custom fields will be stripped

When to use stacked charts:

• Component-level timing breakdowns (e.g., sumcheck, PCS, circuit construction)
• Resource allocation views (e.g., memory by subsystem)
• Any case where you want to see how a total decomposes into parts over time

Adding a New Benchmark

Step 1: Create the Benchmark

TypeScript (most common):

// my_bench.test.ts
import { Timer } from '@aztec/foundation/timer';
import { writeFile, mkdir } from 'fs/promises';

describe('MyComponent benchmarks', () => {
  const results: { name: string; value: number; unit: string }[] = [];

  afterAll(async () => {
    if (process.env.BENCH_OUTPUT) {
      await mkdir(path.dirname(process.env.BENCH_OUTPUT), { recursive: true });
      await writeFile(process.env.BENCH_OUTPUT, JSON.stringify(results));
    }
  });

  it('benchmark operation', async () => {
    const timer = new Timer();
    // ... operation to benchmark ...
    results.push({ name: 'MyComponent/operation/avg', value: timer.ms(), unit: 'ms' });
  });
});

Shell (jq-based):

mkdir -p bench-out
jq -n '[
  {name: "metric1", value: '$VALUE1', unit: "ms"},
  {name: "metric2", value: '$VALUE2', unit: "gas"}
]' > bench-out/my-component.bench.json

Python:

import json
benchmark_list = [{"name": "category/metric", "value": 12345, "unit": "gas"}]
with open("bench-out/my-component.bench.json", "w") as f:
    json.dump(benchmark_list, f)

Step 2: Register in bootstrap.sh

Add to the package's bench_cmds function:

function bench_cmds {
  local hash=$(hash)
  echo "$hash BENCH_OUTPUT=bench-out/my_component.bench.json yarn-project/scripts/run_test.sh <package>/src/my_bench.test.ts"
}

Options: :ISOLATE=1, :CPUS=8, :MEM=16g, :TIMEOUT=7200

CPUS Suggestion: For long running or compute-heavy benchmarks allocate CPUs (:CPUS=N). Benchmarks have strict scheduling, so if you request X CPUs, you'll have them available for consistent results.

ISOLATE Suggestion: Use :ISOLATE=1 when your benchmark needs a clean, isolated environment with no network access and pinned resources. This runs the test in a Docker container, ensuring reproducible results without interference from other processes.

MEM Suggestion: Use :MEM=Xg (e.g., :MEM=16g) for memory-intensive benchmarks that may exceed the default allocation (CPUS × 4GB). Pair with :ISOLATE=1 since memory limits are enforced via Docker.

TIMEOUT Suggestion: Use :TIMEOUT=N (in seconds) for benchmarks that take longer than the default timeout. For example, :TIMEOUT=1800 for 30 minutes, :TIMEOUT=7200 for 2 hours.

Important naming gotcha: Benchmark test files must use .bench.test.ts (with a dot before bench), NOT _bench.test.ts. The test discovery pattern [[ "$test" =~ \.bench\.test\.ts$ ]] specifically looks for .bench.test.ts.

Step 3: Verify

# Run locally
BENCH_OUTPUT=bench-out/test.bench.json yarn test src/my_bench.test.ts

# Validate JSON
jq . bench-out/test.bench.json
jq 'all(has("name") and has("value") and has("unit"))' bench-out/test.bench.json

CI Details

Benchmarks upload when:

• PR has label: ci-merge-queue, ci-full, or ci-full-no-test-cache (publishes to target branch, i.e. next or a merge-train branch)
• Running on merge queue (publishes with next)

10-commit visibility window: The dashboard only shows benchmarks that ran in the last 10 commits. If a benchmark stops running, it disappears after ~10 merges.

Reference Implementations

• TypeScript: yarn-project/stdlib/src/tx/tx_bench.test.ts
• Python: l1-contracts/scripts/generate_benchmark_json.py
• Shell: yarn-project/p2p/testbench/consolidate_benchmarks.sh
• Circuits: noir-projects/noir-protocol-circuits/scripts/run_bench.sh