Bench Compare

Benchmark Comparison Workflow

Compare benchmark results against a baseline to measure optimization impact.

Usage

Invoke with optional benchmark filter:

• /bench-compare - Run all benchmarks
• /bench-compare scan - Run only scan-related benchmarks
• /bench-compare throughput - Run throughput benchmarks

Workflow

1. Establish Baseline

Before making changes, save current benchmark results:

cargo bench --bench <name> -- --save-baseline before

Or for all benchmarks:

cargo bench -- --save-baseline before

2. Apply Changes

Make the optimization changes to the codebase.

3. Compare Results

Run benchmarks against the saved baseline:

cargo bench --bench <name> -- --baseline before

4. Analyze Output

Look for these patterns in Criterion output:

• Performance has improved - Optimization successful
• Performance has regressed - Changes hurt performance
• No change in performance - Within noise threshold

5. Report Summary

Provide a summary table:

Available Benchmarks

The project has these benchmark files in benches/:

• scan.rs - Core scanning performance
• scanner_throughput.rs - End-to-end throughput
• vectorscan_overhead.rs - Vectorscan integration overhead
• rule_isolation.rs - Per-rule cost isolation
• hotspots.rs - Known hot path benchmarks (requires bench feature)
• validator.rs - Validation step benchmarks (requires bench feature)
• Data structure benchmarks: ring_buffer, fixed_set, fixed_vec, timing_wheel, etc.

Tips

• Use --bench <name> to run specific benchmarks for faster iteration
• The hotspots and validator benches require: cargo bench --features bench
• For memory bandwidth tests, use memory_bandwidth.rs
• Compare multiple runs to account for variance