Benchmarking

Benchmarking with cargo xtask bench

The facet project uses a sophisticated benchmarking system that generates Markdown reports comparing performance across multiple targets.

Quick Reference - Running Specific Benchmarks

# Run specific benchmark by name
cargo bench --bench unified_benchmarks_divan -- flatten_2enums

# Run with Tier-2 diagnostics
FACET_TIER2_DIAG=1 cargo bench --bench unified_benchmarks_divan -- flatten_2enums 2>&1 | grep TIER_DIAG

# Check tier2 statistics (attempts/successes/fallbacks)
cargo bench --bench unified_benchmarks_divan -- flatten_2enums 2>&1 | grep TIER_STATS

# Run all benchmarks matching a pattern
cargo bench --bench unified_benchmarks_divan -- flatten

# Run Tier-2 JIT benchmarks only
cargo bench --bench unified_benchmarks_divan -- "tier2"

# List available benchmarks
cargo bench --bench unified_benchmarks_divan -- --list | grep -v "    " | head -20

⚠️ IMPORTANT: Benchmark .rs files are GENERATED from facet-json/benches/benchmarks.kdl. DO NOT edit unified_benchmarks_*.rs directly - edit benchmarks.kdl instead.

Quick Usage

# Run all benchmarks and generate HTML + Markdown report
cargo xtask bench --index --serve

# Run benchmarks without the full perf index (faster)
cargo xtask bench

# Re-analyze existing benchmark data without re-running
cargo xtask bench --no-run

# Run only specific benchmarks (filter passed to cargo bench)
cargo xtask bench --index booleans

# Just generate reports from latest data
cargo xtask bench --no-run --index --serve

How It Works

The benchmarking system has three main components:

1. Benchmark Definition (benchmarks.kdl)

Benchmarks are defined in facet-json/benches/benchmarks.kdl using KDL syntax:

benchmark name="simple_struct" type="SimpleRecord" category="micro" {
    json "{\"id\": 42, \"name\": \"test\", \"active\": true}"
}

benchmark name="booleans" type="Vec<bool>" category="synthetic" {
    generated "booleans"
}

type_def name="SimpleRecord" {
    code """
#[derive(Debug, PartialEq, Facet, serde::Serialize, serde::Deserialize, Clone)]
struct SimpleRecord {
    id: u64,
    name: String,
    active: bool,
}
"""
}

Categories: micro, synthetic, realistic, other Data sources: json (inline), json_file, json_brotli, generated

2. Benchmark Generation (cargo xtask gen-benchmarks)

Run this after editing benchmarks.kdl:

cargo xtask gen-benchmarks

This generates three files in facet-json/:

• benches/unified_benchmarks_divan.rs - Wall-clock timing benchmarks
• benches/unified_benchmarks_gungraun.rs - Instruction count benchmarks
• tests/generated_benchmark_tests.rs - Test versions for valgrind debugging

Every benchmark gets all 4 targets automatically:

• serde_json - Baseline (serde_json crate)
• facet_format_json - facet-format-json without JIT (reflection only)
• facet_format_jit_t1 - Tier-1 JIT (shape-based, ParseEvent stream)
• facet_format_jit_t2 - Tier-2 JIT (format-specific, direct byte parsing)

3. Benchmark Execution and Analysis

cargo xtask bench does:

1. Runs unified_benchmarks_divan (wall-clock times via divan)
2. Runs unified_benchmarks_gungraun (instruction counts via gungraun + valgrind)
3. Parses output and combines results
4. Generates multiple report formats:
   • bench-reports/run.json - Full structured data (schema: run-v1)
   • bench-reports/perf/RESULTS.md - Markdown report for LLMs and humans
   • bench-reports/perf-data.json - Legacy format for perf tracking

The Markdown Report (perf/RESULTS.md)

Located at bench-reports/perf/RESULTS.md, this is the authoritative source for performance analysis:

Structure:

• Targets table - Definitions of all benchmark targets
• Benchmark sections - Grouped by category (Micro, Synthetic, Realistic)
• Per-benchmark tables - Deserialize and Serialize results
  • Columns: Target, Time (median), Instructions, vs serde_json ratio
  • Ratios: **0.84×** ✓ (wins), 1.03× (close), 3.12× ⚠ (needs work)
• Summary - Auto-categorized by performance:
  • Wins: ≤1.0× vs serde_json
  • Close: ≤1.5× vs serde_json
  • Needs Work: >1.5× vs serde_json

Example:

### booleans

**Deserialize:**

| Target | Time (median) | Instructions | vs serde_json |
|--------|---------------|--------------|---------------|
| serde_json | 56.21µs | 1,157,922 | 1.00× |
| format+jit2 | 53.46µs | 972,221 | **0.84×** ✓ |
| format+jit1 | 809.30µs | 7,031,459 | 6.07× ⚠ |
| format | 2.94ms | 23,169,951 | 20.01× ⚠ |

Adding New Benchmarks

1. Edit facet-json/benches/benchmarks.kdl

   benchmark name="my_bench" type="MyType" category="synthetic" {
       generated "my_generator"
   }
   
   type_def name="MyType" {
       code """
   #[derive(Debug, Facet, serde::Serialize, serde::Deserialize, Clone)]
   struct MyType {
       field: String,
   }
   """
   }
   

2. If using generated, add generator to tools/benchmark-generator/src/main.rs

   • Edit generate_json_data() function
   • Add case for your generator name

3. Regenerate benchmarks

   cargo xtask gen-benchmarks
   

4. Run benchmarks

   cargo xtask bench --index --serve
   

Important Flags

--no-run

Skips running benchmarks, uses latest data. Useful for:

• Regenerating reports after fixing parser bugs
• Testing report generation changes
• Quick iterations on report formatting

--index

Generates the full perf.facet.rs index:

• Clones the facet-rs/perf.facet.rs repo (gh-pages branch)
• Copies benchmark reports to bench-reports/perf/
• Generates index.html and supporting files
• Required for viewing the interactive SPA

--serve

Starts a local server at http://localhost:1999 to view reports. Requires --index.

--push

Pushes generated reports to the perf.facet.rs repo. Use with caution - only for publishing official results.

Debugging Benchmarks with Valgrind

The generated tests in tests/generated_benchmark_tests.rs mirror the benchmarks and can be run under valgrind:

# Run specific benchmark as test under valgrind
cargo nextest run --profile valgrind -p facet-json generated_benchmark_tests::test_booleans --features jit

# Or use the generated test filters
cargo nextest run --profile valgrind -p facet-json test_simple_struct --features jit

This is essential for debugging crashes or memory issues in benchmarks.

Files and Directories

bench-reports/
├── divan-{timestamp}.txt          # Raw divan output
├── gungraun-{timestamp}.txt       # Raw gungraun output
├── run.json                       # Structured results (run-v1 schema)
├── perf-data.json                 # Legacy perf tracking format
└── perf/
    ├── RESULTS.md                 # **MAIN REPORT - READ THIS**
    ├── index.html                 # SPA (generated with --index)
    ├── app.js                     # SPA logic (copied from scripts/)
    └── shared-styles.css          # SPA styles (copied from scripts/)

facet-json/benches/
├── benchmarks.kdl                 # **EDIT THIS to add benchmarks**
├── unified_benchmarks_divan.rs    # Generated (divan)
└── unified_benchmarks_gungraun.rs # Generated (gungraun)

facet-json/tests/
└── generated_benchmark_tests.rs   # Generated (for valgrind)

tools/
├── benchmark-generator/           # KDL → Rust codegen
└── benchmark-analyzer/            # Output parsing + report generation

Don't Edit Generated Files

❌ NEVER edit these files (they're regenerated):

• unified_benchmarks_divan.rs
• unified_benchmarks_gungraun.rs
• generated_benchmark_tests.rs
• bench-reports/perf/index.html, app.js, shared-styles.css

✅ Edit these instead:

• facet-json/benches/benchmarks.kdl - Benchmark definitions
• tools/benchmark-generator/src/main.rs - Generator logic (for generated benchmarks)
• scripts/app.js, scripts/shared-styles.css - SPA source (not the copies in perf/)

Common Workflows

Quick local benchmark run

cargo xtask bench
# Check bench-reports/perf/RESULTS.md

Full interactive report

cargo xtask bench --index --serve
# Opens http://localhost:1999

After editing benchmarks.kdl

cargo xtask gen-benchmarks
cargo xtask bench

Re-analyze existing data

cargo xtask bench --no-run --index

Benchmark a specific test

cargo xtask bench integers
# Only runs benchmarks matching "integers"

Performance Analysis Tips

1. Focus on the Markdown report first (perf/RESULTS.md)

   • Easy to grep, parse, and read
   • Shows all critical metrics in one place
   • Auto-categorized by performance tier

2. Use instruction counts, not just time

   • More stable than wall-clock time
   • Architecture-independent
   • Appears in "vs serde_json" column when available

3. Look for patterns in the Summary section

   • "Needs Work" items are optimization targets
   • "Wins" validate current approach
   • "Close" items are low-hanging fruit

4. Compare Tier-1 vs Tier-2 JIT

   • Large gaps = Tier-2 not implemented or buggy
   • Similar performance = Tier-2 working but not optimized
   • Tier-2 wins = format-specific optimizations paying off

Troubleshooting

Benchmarks fail to compile

# Regenerate from KDL
cargo xtask gen-benchmarks

Parser errors in output

• Check bench-reports/divan-*.txt or gungraun-*.txt for malformed output
• Fix the benchmark code, not the parser (usually)

Missing benchmarks in report

• Ensure benchmark has category in benchmarks.kdl
• Check that cargo xtask gen-benchmarks ran successfully
• Verify benchmark functions are generated (check unified_benchmarks_*.rs)

--index fails

• Ensure gh CLI is installed and authenticated
• Check network connection (clones from GitHub)
• Try --index without --push first

See Also

• divan docs: https://docs.rs/divan/
• gungraun: Custom fork with valgrind integration
• Nextest valgrind profile: .config/nextest.toml
• Benchmark generator: tools/benchmark-generator/
• Report analyzer: tools/benchmark-analyzer/