Haskell Benchmarking

Haskell Benchmarking

Default stack: tasty-bench for timing + weigh for allocation. Use criterion only when you specifically need its HTML reports.

Benchmarks live in a dedicated benchmark cabal stanza, separate from tests, and are run with cabal bench (not cabal test). Don't mix the two — tests are correctness gates that run on every commit; benchmarks are observability tools that run on demand or nightly because their variance is too high to gate builds.

Cabal stanza

Place benchmarks in bench/ next to src/ and test/. Compile with -O2 always — benchmarking -O0 code measures nothing useful.

benchmark my-project-bench
  import:           shared
  type:             exitcode-stdio-1.0
  hs-source-dirs:   bench
  main-is:          Bench.hs
  ghc-options:      -O2
  build-depends:
    , base
    , my-project
    , tasty-bench  ^>=0.4
    , weigh        ^>=0.0.17

Run with cabal bench (or cabal bench my-project-bench to target a specific stanza).

Timing benchmarks with tasty-bench

module Main (main) where

import Test.Tasty.Bench
import qualified Data.Map.Strict as Map
import qualified Data.IntMap.Strict as IntMap

main :: IO ()
main = defaultMain
  [ bgroup "lookup"
      [ bench "Map.lookup    @1000"  $ nf (Map.lookup    500)    mapData
      , bench "IntMap.lookup @1000"  $ nf (IntMap.lookup 500)    intMapData
      ]
  , bgroup "insert"
      [ bench "Map.insert    @1000"  $ nf (Map.insert    1001 ()) mapData
      , bench "IntMap.insert @1000"  $ nf (IntMap.insert 1001 ()) intMapData
      ]
  ]
  where
    mapData    = Map.fromList    [(i, ()) | i <- [1..1000]]
    intMapData = IntMap.fromList [(i, ()) | i <- [1..1000]]

Key functions:

• bench "name" $ nf f x — runs f x and forces the result to normal form (fully evaluated). Use this for pure functions returning data structures.
• bench "name" $ whnf f x — forces to weak head normal form (outermost constructor only). Use when nf would do more work than the function itself.
• bgroup "label" — nests benchmarks for organization.
• env setup $ \resource -> bench ... — for benchmarks that need expensive setup that shouldn't count toward the timing.

The laziness trap

This is the #1 mistake. bench "foo" $ whnf id heavyComputation measures nothing useful — id returns a thunk untouched. Always pick nf/whnf based on what the function actually needs to compute:

• Returning Int, Bool, a small atom → whnf is enough.
• Returning Map, [a], a record → nf to force the whole structure.
• Returning a partial function or curried result → split the call so the benchmark measures application, not partial application.

If you're not sure, use nf. Over-forcing makes benchmarks slower but never wrong; under-forcing makes them lie.

Tracking regressions with baselines

# Save current numbers as the baseline.
cabal bench --benchmark-options="--csv baseline.csv"

# After a change, compare.
cabal bench --benchmark-options="--baseline baseline.csv --fail-if-slower 10"

--fail-if-slower 10 exits non-zero if any benchmark got >10% slower vs. baseline. This is what makes benchmarks useful in CI — not absolute numbers, but deltas. See the CI section below.

Allocation benchmarks with weigh

weigh answers a different question: how much memory does this allocate? Allocation count is a leading indicator of space leaks and tends to be more stable across machines than wall-clock time.

module Main (main) where

import Weigh
import qualified Data.Map.Strict as Map
import qualified Data.IntMap.Strict as IntMap

main :: IO ()
main = mainWith $ do
  func "Map.fromList    @1000"   Map.fromList    [(i, ()) | i <- [1..1000]]
  func "IntMap.fromList @1000"   IntMap.fromList [(i, ()) | i <- [1..1000]]

  -- For values (not functions), use `value`:
  value "thunked 1M Ints" [1..1000000 :: Int]

Output is a tabular allocation report (bytes allocated, GCs, max residency). A function that allocates 10× more than its replacement is a regression worth investigating even if wall-clock times look similar — the cost shows up later as cache pressure or GC pauses.

Pair tasty-bench and weigh in the same benchmark stanza by putting both in Bench.hs (calling each in its own section), or split them into two stanzas if you want to run them independently.

When to switch to criterion

tasty-bench covers ~95% of what criterion does, with a near-identical API. Reach for criterion when:

• You want the HTML report with interactive plots (variance, confidence intervals, regression plots). Run cabal bench -- --output report.html.
• You need OLS regression to decompose constant cost from per-element cost across input sizes.
• You're publishing benchmark results externally and want richer presentation.

Migration is cheap because the function names match. Swap the cabal dep and the import:

build-depends:
  , criterion    ^>=1.6   -- was: tasty-bench

import Criterion.Main     -- was: Test.Tasty.Bench

The bench, bgroup, nf, whnf, env, defaultMain API is the same. Only difference users tend to hit: criterion doesn't accept --baseline/--fail-if-slower directly (it has its own JSON output flow), so the CI regression workflow needs a different shape.

If you're unsure, stay on tasty-bench. The HTML report is rarely worth the build-time and dependency cost.

Going deeper: GHC profiling

Benchmarks tell you what is slow. GHC's profiling tools tell you why. When a benchmark surfaces a regression you don't understand:

ghc-options: -O2 -prof -fprof-auto -fprof-cafs

Then:

cabal run my-project-bench -- +RTS -p          # cost-center profile
cabal run my-project-bench -- +RTS -hc -i0.1   # heap profile by cost center
cabal run my-project-bench -- +RTS -s          # GC and allocation summary

Visualize with:

• ghc-prof-flamegraph — converts .prof to flame graphs (hot-path visualization).
• hp2pretty — turns .hp heap profiles into SVG. Watch for thunks accumulating over time → space leak.
• threadscope + -eventlog flag — for concurrency profiling (async, STM, sparks).

This is the same toolset mentioned briefly in haskell-quality-gates under "Performance Defaults"; use this skill when you need the full benchmark-driven workflow, that one for the smaller "use Text not String" hygiene rules.

Conventions

• Benchmark files mirror source files, like spec files: src/MyProject/Codec.hs ↔ bench/MyProject/CodecBench.hs. The single Bench.hs main-is file imports each and assembles them under defaultMain.
• Benchmark inputs in the file, not generated at runtime, when feasible. Reproducibility matters more than coverage.
• Don't benchmark IO actions unless the IO is the point. Pure functions are what tasty-bench and weigh measure cleanly; for IO performance, you usually want a load-testing tool, not a microbenchmark.
• Strict by default — see the strict-fields guidance in haskell-quality-gates. Lazy fields are the #1 cause of "this got slower after a refactor."

CI considerations

Benchmarks don't belong in the normal CI path that runs on every push — variance from shared runners makes the numbers noisy and the false-positive rate too high.

Two patterns that work:

1. Nightly benchmark job, separate from PR CI, that runs benchmarks against main and stores the CSV. Open an issue automatically if any benchmark regresses >X% vs. last week's baseline.
2. On-demand benchmark comparison on PRs, gated by a label or comment (e.g., /benchmark), that runs benchmarks on the PR branch and main back-to-back on the same runner and posts the delta.

Both approaches sidestep the "noisy CI" problem by comparing two runs on the same hardware in the same job, not absolute numbers.

Related

• Cabal stanza shape and common shared block: haskell-project-setup.
• Strict fields, foldl', +RTS -s, and the rest of the "performance defaults" hygiene that prevents regressions in the first place: haskell-quality-gates.
• Why tests and benchmarks live in different stanzas: tests are correctness gates (cabal test), benchmarks are observability tools (cabal bench). See haskell-testing for the test side.