Anomalib Benchmarking

Using the Benchmarking Pipeline

The benchmarking pipeline runs a grid of model/dataset/category combinations end-to-end (train + test) and writes measured metrics to a CSV — use it to produce real, reproducible numbers rather than hand-editing benchmark tables.

Code locations

• src/anomalib/pipelines/benchmark/pipeline.py — Benchmark: top-level pipeline; picks SerialRunner or ParallelRunner based on configured accelerators and torch.cuda.device_count().
• src/anomalib/pipelines/benchmark/generator.py — BenchmarkJobGenerator: expands the config (including grid: entries) into individual jobs.
• src/anomalib/pipelines/benchmark/job.py — BenchmarkJob: runs one model/dataset combination, times it, and saves results.
• tools/experimental/benchmarking/benchmark.py — thin CLI wrapper around Benchmark.
• tools/experimental/benchmarking/sample.yaml — example config to copy from.

Running it

# Via the tools wrapper
python tools/experimental/benchmarking/benchmark.py --config tools/experimental/benchmarking/sample.yaml

# Via the anomalib CLI (registered pipeline subcommand)
anomalib benchmark --config tools/experimental/benchmarking/sample.yaml

Config structure

accelerator:
  - cuda
  - cpu

benchmark:
  seed: 42
  model:
    class_path:
      grid: [Padim, Patchcore]
  data:
    class_path: MVTecAD
    init_args:
      category:
        grid:
          - bottle
          - capsule

Any field can use grid: [...] to sweep multiple values — the generator produces the Cartesian product of every grid field as separate jobs (here: 2 models × 2 categories = 4 jobs). Non-grid fields are held constant across all jobs. data.class_path / model.class_path follow the same anomalib.data.* / anomalib.models.* resolution as everywhere else in the repo (see anomalib-training).

Where results go

BenchmarkJob.save(...) writes one row per job into:

runs/benchmark/<timestamp>/results.csv

(<timestamp> is generated when results are saved via BenchmarkJob.save(), e.g. 2026-08-24-10_30_00.) Each row includes the model/dataset/category combination and the measured metrics — this is the file to consume when building or refreshing README/docs benchmark tables.

There is also a separate, narrower helper tools/benchmark_mebin.py that writes to results/mebin_benchmark.csv for a specific benchmarking use case — prefer the pipeline above unless you specifically need that script's behavior.

Gotchas

• A grid sweep multiplies job count fast — check the Cartesian product size before launching a large sweep (e.g. 5 models × 10 categories = 50 full train+test runs).
• accelerator: [cuda, cpu] creates one runner per entry, so every model/category combination runs once per accelerator (doubling the total job count). This is not a device-pool selector — if you only want to benchmark on GPU, use accelerator: [cuda].
• Never hand-write or infer numbers into README/docs benchmark tables — always source them from a results.csv produced by an actual run of this pipeline.

Reviewer / self-check

• Config's grid fields produce the intended, bounded set of jobs (no accidental huge sweep).
• model.class_path / data.class_path values resolve to real exported classes.
• Benchmark run completed and runs/benchmark/<timestamp>/results.csv exists before citing numbers anywhere else.
• Test reference: tests/integration/pipelines/test_benchmark.py for how the pipeline is invoked programmatically if debugging job generation.