Benchmark

Benchmark

Run benchmarks using the release build. Build first with /build.

Usage

• /benchmark asr — full WER + RTF + peakRSS + throughput on a labeled dataset (LibriSpeech-style dir or .tsv manifest). Extra args pass through to asr-bench. Requires BENCH_DATASET env var or --dataset <path> in the trailing args.
• /benchmark asr-quick — single-file RTF smoke test (no WER, no dataset required). Equivalent to the old asr behavior.
• /benchmark tts — synthesize test text, report RTF
• /benchmark vad — VAD on VoxConverse (all engines)
• /benchmark diarize — DER on VoxConverse (requires downloaded test set)

Examples

Compare our MLX quantizations on LibriSpeech test-clean (WER + RTF + peakRSS, each engine isolated in its own process so RSS reflects per-engine cost):

BENCH_DATASET=$HOME/datasets/LibriSpeech/test-clean /benchmark asr \
  --engines qwen3-mlx-0.6b-4bit qwen3-mlx-0.6b-8bit \
  --isolated --limit 50

Default engine set (qwen3-coreml + parakeet + whisperkit) on a TSV manifest:

/benchmark asr --dataset bench.tsv --limit 100 --output /tmp/run.json

module="$1"
shift || true
cli=".build/release/speech"
bench=".build/release/asr-bench"

case "$module" in
  asr)
    if [ ! -x "$bench" ]; then
      echo "asr-bench binary missing — run /build first (release)." >&2
      exit 1
    fi
    # Honor BENCH_DATASET if --dataset isn't already in the trailing args.
    has_dataset=0
    for a in "$@"; do
      if [ "$a" = "--dataset" ]; then has_dataset=1; break; fi
    done
    if [ "$has_dataset" = "0" ] && [ -n "$BENCH_DATASET" ]; then
      set -- --dataset "$BENCH_DATASET" "$@"
    fi
    "$bench" "$@" 2>&1
    ;;
  asr-quick)
    $cli transcribe Tests/Qwen3ASRTests/Resources/test_audio.wav 2>&1
    ;;
  tts)
    $cli speak "The quick brown fox jumps over the lazy dog." --output /tmp/bench_tts.wav 2>&1
    ;;
  vad)
    python3 scripts/benchmark_vad.py --compare --num-files 5 2>&1
    ;;
  diarize)
    python3 scripts/benchmark_diarization.py --num-files 5 2>&1
    ;;
  *)
    echo "Usage: /benchmark [asr|asr-quick|tts|vad|diarize] [args...]"
    echo "  asr       — full WER + RTF + peakRSS via asr-bench (needs dataset)"
    echo "  asr-quick — single-file RTF smoke test (no dataset)"
    ;;
esac

What /benchmark asr reports

Per engine, in the printed table and the JSON output:

Use --isolated to run each engine in a child process. RSS and physical- footprint high-water marks then reflect one engine instead of the cumulative state of a sequential multi-engine run.

Available engines

qwen3-coreml, qwen3-mlx-{0.6b,1.7b}-{4bit,8bit}, parakeet, nemotron, nemotron-mlx-{int5,int8}, omnilingual, omnilingual-mlx-{300m,1b,3b,7b}-4bit, whisperkit-{large-v3-turbo,large-v3,distil-large-v3}.

Performance targets (M2 Max)