# Haskell Benchmarking

> Write and run Haskell performance benchmarks. Covers the `benchmark` cabal stanza, `tasty-bench` for timing (the default), `weigh` for memory-allocation measurement, baseline comparison to catch regressions, the laziness traps (`nf` vs `whnf`), and how to integrate with GHC profiling/flame graphs for deeper investigation. Also explains when to switch to `criterion` for HTML reports. Use when adding benchmarks, debugging a "this got slower" regression, comparing two implementations of the same function, hunting space leaks, or wiring benchmarks into CI.

- Skill: `ivelten/haskell-benchmarking` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ivelten/haskell-benchmarking`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ivelten/haskell-benchmarking/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: ivelten (https://skillmd.com/u/ivelten)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/ivelten/haskell-benchmarking

---


# Haskell Benchmarking

Default stack: **`tasty-bench`** for timing + **`weigh`** for allocation. Use `criterion` only when you specifically need its HTML reports.

Benchmarks live in a dedicated `benchmark` cabal stanza, separate from tests, and are run with `cabal bench` (not `cabal test`). Don't mix the two — tests are correctness gates that run on every commit; benchmarks are observability tools that run on demand or nightly because their variance is too high to gate builds.

## Cabal stanza

Place benchmarks in `bench/` next to `src/` and `test/`. Compile with `-O2` always — benchmarking `-O0` code measures nothing useful.

```cabal
benchmark my-project-bench
  import:           shared
  type:             exitcode-stdio-1.0
  hs-source-dirs:   bench
  main-is:          Bench.hs
  ghc-options:      -O2
  build-depends:
    , base
    , my-project
    , tasty-bench  ^>=0.4
    , weigh        ^>=0.0.17
```

Run with `cabal bench` (or `cabal bench my-project-bench` to target a specific stanza).

## Timing benchmarks with `tasty-bench`

```haskell
module Main (main) where

import Test.Tasty.Bench
import qualified Data.Map.Strict as Map
import qualified Data.IntMap.Strict as IntMap

main :: IO ()
main = defaultMain
  [ bgroup "lookup"
      [ bench "Map.lookup    @1000"  $ nf (Map.lookup    500)    mapData
      , bench "IntMap.lookup @1000"  $ nf (IntMap.lookup 500)    intMapData
      ]
  , bgroup "insert"
      [ bench "Map.insert    @1000"  $ nf (Map.insert    1001 ()) mapData
      , bench "IntMap.insert @1000"  $ nf (IntMap.insert 1001 ()) intMapData
      ]
  ]
  where
    mapData    = Map.fromList    [(i, ()) | i <- [1..1000]]
    intMapData = IntMap.fromList [(i, ()) | i <- [1..1000]]
```

Key functions:

- **`bench "name" $ nf f x`** — runs `f x` and forces the result to **normal form** (fully evaluated). Use this for pure functions returning data structures.
- **`bench "name" $ whnf f x`** — forces to **weak head normal form** (outermost constructor only). Use when `nf` would do more work than the function itself.
- **`bgroup "label"`** — nests benchmarks for organization.
- **`env setup $ \resource -> bench ...`** — for benchmarks that need expensive setup that shouldn't count toward the timing.

### The laziness trap

This is the #1 mistake. `bench "foo" $ whnf id heavyComputation` measures nothing useful — `id` returns a thunk untouched. Always pick `nf`/`whnf` based on what the function actually needs to compute:

- Returning `Int`, `Bool`, a small atom → `whnf` is enough.
- Returning `Map`, `[a]`, a record → `nf` to force the whole structure.
- Returning a partial function or curried result → split the call so the benchmark measures application, not partial application.

If you're not sure, use `nf`. Over-forcing makes benchmarks slower but never wrong; under-forcing makes them lie.

### Tracking regressions with baselines

```bash
# Save current numbers as the baseline.
cabal bench --benchmark-options="--csv baseline.csv"

# After a change, compare.
cabal bench --benchmark-options="--baseline baseline.csv --fail-if-slower 10"
```

`--fail-if-slower 10` exits non-zero if any benchmark got >10% slower vs. baseline. This is what makes benchmarks useful in CI — not absolute numbers, but deltas. See the CI section below.

## Allocation benchmarks with `weigh`

`weigh` answers a different question: **how much memory does this allocate?** Allocation count is a leading indicator of space leaks and tends to be more stable across machines than wall-clock time.

```haskell
module Main (main) where

import Weigh
import qualified Data.Map.Strict as Map
import qualified Data.IntMap.Strict as IntMap

main :: IO ()
main = mainWith $ do
  func "Map.fromList    @1000"   Map.fromList    [(i, ()) | i <- [1..1000]]
  func "IntMap.fromList @1000"   IntMap.fromList [(i, ()) | i <- [1..1000]]

  -- For values (not functions), use `value`:
  value "thunked 1M Ints" [1..1000000 :: Int]
```

Output is a tabular allocation report (bytes allocated, GCs, max residency). A function that allocates 10× more than its replacement is a regression worth investigating even if wall-clock times look similar — the cost shows up later as cache pressure or GC pauses.

Pair `tasty-bench` and `weigh` in the **same** `benchmark` stanza by putting both in `Bench.hs` (calling each in its own section), or split them into two stanzas if you want to run them independently.

## When to switch to `criterion`

`tasty-bench` covers ~95% of what `criterion` does, with a near-identical API. Reach for `criterion` when:

- You want the **HTML report** with interactive plots (variance, confidence intervals, regression plots). Run `cabal bench -- --output report.html`.
- You need **OLS regression** to decompose constant cost from per-element cost across input sizes.
- You're publishing benchmark results externally and want richer presentation.

Migration is cheap because the function names match. Swap the cabal dep and the import:

```cabal
build-depends:
  , criterion    ^>=1.6   -- was: tasty-bench
```

```haskell
import Criterion.Main     -- was: Test.Tasty.Bench
```

The `bench`, `bgroup`, `nf`, `whnf`, `env`, `defaultMain` API is the same. Only difference users tend to hit: `criterion` doesn't accept `--baseline`/`--fail-if-slower` directly (it has its own JSON output flow), so the CI regression workflow needs a different shape.

If you're unsure, **stay on tasty-bench**. The HTML report is rarely worth the build-time and dependency cost.

## Going deeper: GHC profiling

Benchmarks tell you *what* is slow. GHC's profiling tools tell you *why*. When a benchmark surfaces a regression you don't understand:

```cabal
ghc-options: -O2 -prof -fprof-auto -fprof-cafs
```

Then:

```bash
cabal run my-project-bench -- +RTS -p          # cost-center profile
cabal run my-project-bench -- +RTS -hc -i0.1   # heap profile by cost center
cabal run my-project-bench -- +RTS -s          # GC and allocation summary
```

Visualize with:

- **`ghc-prof-flamegraph`** — converts `.prof` to flame graphs (hot-path visualization).
- **`hp2pretty`** — turns `.hp` heap profiles into SVG. Watch for thunks accumulating over time → space leak.
- **`threadscope`** + `-eventlog` flag — for concurrency profiling (`async`, STM, sparks).

This is the same toolset mentioned briefly in `haskell-quality-gates` under "Performance Defaults"; use this skill when you need the full benchmark-driven workflow, that one for the smaller "use Text not String" hygiene rules.

## Conventions

- **Benchmark files mirror source files**, like spec files: `src/MyProject/Codec.hs` ↔ `bench/MyProject/CodecBench.hs`. The single `Bench.hs` `main-is` file imports each and assembles them under `defaultMain`.
- **Benchmark inputs in the file, not generated at runtime**, when feasible. Reproducibility matters more than coverage.
- **Don't benchmark `IO` actions** unless the IO is the point. Pure functions are what `tasty-bench` and `weigh` measure cleanly; for IO performance, you usually want a load-testing tool, not a microbenchmark.
- **Strict by default** — see the strict-fields guidance in `haskell-quality-gates`. Lazy fields are the #1 cause of "this got slower after a refactor."

## CI considerations

Benchmarks don't belong in the normal CI path that runs on every push — variance from shared runners makes the numbers noisy and the false-positive rate too high.

Two patterns that work:

1. **Nightly benchmark job**, separate from PR CI, that runs benchmarks against `main` and stores the CSV. Open an issue automatically if any benchmark regresses >X% vs. last week's baseline.
2. **On-demand benchmark comparison on PRs**, gated by a label or comment (e.g., `/benchmark`), that runs benchmarks on the PR branch and `main` back-to-back on the same runner and posts the delta.

Both approaches sidestep the "noisy CI" problem by comparing two runs on the same hardware in the same job, not absolute numbers.

## Related

- Cabal stanza shape and `common shared` block: `haskell-project-setup`.
- Strict fields, `foldl'`, `+RTS -s`, and the rest of the "performance defaults" hygiene that prevents regressions in the first place: `haskell-quality-gates`.
- Why tests and benchmarks live in different stanzas: tests are correctness gates (`cabal test`), benchmarks are observability tools (`cabal bench`). See `haskell-testing` for the test side.