# Xtdb Testing

> XTDB-specific rules and mechanics for running tests — running the task yourself, delegating the failure text, the mid-run edit freeze, test tasks and filters, iteration counts, simulation-test coverage, diagnosing failures, and regenerating arrow-edn golden fixtures. Read this before running or delegating any test run in this repo.

- Skill: `xtdb/xtdb-testing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add xtdb/xtdb-testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/xtdb/xtdb-testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: xtdb (https://skillmd.com/u/xtdb)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/xtdb/xtdb-testing

---


# Running tests in XTDB

Read this before you run a test.

You run the test task yourself and redirect Gradle's output to a log.
`gradle-test-results` then reads that log and tells you what happened — delegation is for the reading, not the running.

Interpret MUST, MUST NOT, SHOULD, SHOULD NOT, MAY per RFC 2119.

## The rules you MUST NOT get wrong

1. You MUST run the test task yourself, redirecting its output to `build/test-run.log` — never delegate the run, and never let a test run's output into your context.
2. You MUST hand that log to `gradle-test-results` rather than reading it yourself — see [Delegating the reading](#delegating-the-reading).
3. You MUST NOT edit files the build compiles while your run is compiling them — see [The build phase is what is frozen](#the-build-phase-is-what-is-frozen).
4. A test that fails after your change is a test *you* broke — see [When a test fails](#when-a-test-fails).
5. You MUST stop a run the moment you know you'll re-run it, rather than letting it finish — see [A run you already know you'll redo is waste](#a-run-you-already-know-youll-redo-is-waste).

The rest of this document is the mechanics behind those five.

## The loop

Two commands, from the project root:

```sh
./gradlew :testClasses                                               # compile — see the edit freeze below
mkdir -p build; ./gradlew :test --tests 'xtdb.api_test*' > build/test-run.log 2>&1
```

Then hand the log to `gradle-test-results`:

> Read `/abs/path/to/tree/build/test-run.log` and the result files under `/abs/path/to/tree`, and report what happened.
> `./gradlew :test --tests 'xtdb.api_test*'` exited N.

- **You SHOULD run the test command with `run_in_background: true`** for anything longer than a single namespace, and get on with work the freeze doesn't cover while it runs.
- **You MUST NOT run more than one `./gradlew` invocation at a time in a worktree** — concurrent invocations can corrupt the build cache, particularly with Kotlin.
  Combine every namespace you want covered into a single invocation (`--tests '*foo*' --tests '*bar*'`) and let Gradle parallelise internally.

### Delegating the reading

`gradle-test-results` is defined in `.claude/agents/`, and holds no `./gradlew`, no `Edit` and no `Write`.

**Give it the tree as an absolute path.**
It inherits your working directory, the Task tool has no parameter that pins it, and worktrees here live under `.claude/worktrees/` — *inside* the main checkout — so an agent in the wrong tree finds a plausible set of result files rather than an error.

Whether a failure it reports is yours, pre-existing or a known flake is [yours to judge](#when-a-test-fails).

### Keeping the output out of your context

A bare `./gradlew test` puts megabytes of task progress, stack traces and daemon chatter into your context.

- You MUST NOT read the log of a run that reached the tests — not with `Read`, not with `cat`, `tail` or `grep`.
- **A compile or configuration failure is the exception**: nothing ran, so the log is a couple of hundred lines with the error in the last thirty of them.
  `tail -30 build/test-run.log` and fix it, rather than paying a delegation round-trip to be told what the compiler already said.
- One path, overwritten by each run, so there is only ever one file to name and no chance of reading the last run's.
- `build/` is gitignored, so the log never shows up in `git status`, and `clean` disposes of it.

## Running the tests has no legitimate delegate

- You MUST NOT use `repl-explorer` to run tests.
  Its own workflow tells it to "modify code and reload" when a test fails, so it edits source on your behalf and reports green.
  The `/clojure-eval` skill and `clj-nrepl-eval` remain the right tools for *exploratory* evaluation — inspecting state, trying an expression, reproducing a bug.
  They are not the tool for "run the tests and tell me whether they pass".

You SHOULD run the relevant tests proactively after a code change rather than waiting to be asked.

## A comment-only change needs a compile, not a test run

"Rerun after every change, however trivial" is about changes that can alter behaviour.
A change confined to comments, KDoc or Javadoc cannot, so a clean `./gradlew :testClasses` (plus `:xtdb-core:testClasses` for a core change) is the whole of the verification it needs, and you MUST NOT spend a suite run on it.

The carve-out is narrow, and these are the ways a "comment change" turns out not to be one:

- **An annotation is not a comment.**
  `@Test`, `@Tag`, `@OptIn`, `@JvmStatic`, `@Suppress` all change what the compiler or the runner does.
- **A Clojure `;;` line is a comment; `(comment ...)` and `#_` are not.**
  Both are read by the reader, and `#_` elides the *next form* — moving one changes what compiles.
- **A docstring is a value, not a comment.**
  Editing one changes the compiled var, which is why it takes the same care as code even though nothing observable usually depends on it.
- **A doc file the build reads is not a comment either.**
  `.allium`, `.adoc` and `README`s are outside the build; anything on the test resource path is not.

Where a diff is comments *plus* code, it is a code change — run the tests.

## A run you already know you'll redo is waste

A run only ever describes the tree it compiled.
So the moment you decide on a material change — a bug you spotted reviewing the diff, a fix for a failure the run has already reported, anything at all that touches a file the build reads — that run's verdict is void, and every remaining minute of it buys nothing.

Kill it and re-run with the change in.

- You MUST NOT let a run you have already invalidated play out on the grounds that its remaining results might still be worth having.
  They are not reportable: you cannot claim a namespace passed in a tree you are about to change, and sorting the failures that survive your edit from the ones that don't costs more than the re-run.
- Reading the diff while a run is in flight is the right use of the wait, and finding something is the expected outcome.
  A finding is a reason to stop the run — not something to sit on until it finishes.
- The run is your own process, so stopping it is yours to do: `TaskStop` on a backgrounded run, or interrupt the foreground call.
- Stopping the run is also what lifts the edit freeze, so the order is: stop it, confirm no test worker is still up, then edit.

## The build phase is what is frozen

The freeze is scoped to the worktree the run is compiling from, and within it to **the files that run compiles** — Kotlin, Java and Clojure sources, build scripts, and anything on the test resource path.
Recompiling under a running build produces **bogus cross-language type errors and cascading failures** that look exactly like real breakage, and chasing them costs far more than waiting did.
If you have already edited mid-compile, discard that run's results entirely and re-run once the tree is stable — do not try to reason about which failures were real.

Compilation is the part a concurrent edit corrupts, so step 1 of [the loop](#the-loop) exists to get it over with while you are still waiting anyway:

1. `./gradlew :testClasses`, or `:xtdb-core:testClasses` for a module run.
   That is the entire compile phase behind the root `test` task: every module's `compileKotlin` and `jar`, plus `compileTestClojure` and `compileTestFixturesClojure`, which AOT-compile the Clojure test and fixture namespaces into `build/clojure/`.
   Its output is small enough to read directly, so it needs no redirect and no delegate — but the freeze applies while it is in flight.
2. The test command. Its compile tasks are up-to-date and clear in a few seconds on a warm daemon, and nothing is compiled after that.
3. Edit from that point on.

Unsplit, the freeze covers however long the run takes to build — minutes, after a Kotlin change or a cold `build/`.
Split, it is the few seconds at the top of the test invocation.

Two things the split does not buy:

- **The report describes the tree you compiled, not the tree you now have.**
  Anything edited after step 1 is untested until the next run, whatever the report says, so you MUST NOT report green for a file you edited during the run that produced it.
  The split lets you get on with the *next* increment while a run confirms the last one; it does not let you fix a failure and claim the same run vindicates the fix.
- **A module test run keeps that module's `src/main/clojure` live on the classpath.**
  A Clojure source set that isn't AOT-compiled has its source directory as its `sourceSet.output`, so `:xtdb-core:test` loads `core/src/main/clojure` from source as each namespace is first required.
  An edit therefore lands in the running JVM and the run executes a mixture of old and new — silently, with no compile error to give it away.
  Root `:test` is not exposed to this, because module Clojure reaches it through the jars built in step 1 and its own test and fixture namespaces are AOT-compiled.
  So `.clj` edits stay frozen for the duration of a module run, however you split it.

Everything the build never reads is outside the freeze, and you SHOULD carry on with it rather than idling: `.allium` specs, `docs/`, `dev/`, `README`s, `.claude/`.
A comment-only change still counts as a source edit — the compiler doesn't know it was only a comment.

Runs in other worktrees do not concern you, and you MUST NOT check for them or wait on them.
Each worktree has its own `build/` and `.gradle/`, and Gradle takes cross-process locks over the shared `~/.gradle` caches, so a build elsewhere on the machine cannot corrupt yours.
Those locks block rather than fail, so a run that stalls early — typically reporting that it is waiting to acquire a lock — is that mechanism working; wait it out rather than killing the run.

## Test tasks

- `./gradlew test` — unit tests.
  Excludes the `integration`, `property`, `jdbc`, `timescale`, `s3`, `minio`, `slt`, `docker`, `azure` and `google-cloud` tags.
- `./gradlew integration-test` — integration tests, longer running.
- `./gradlew property-test` — property-based and simulation tests.
- `./gradlew kafka-test` — tests needing Kafka; requires `docker-compose up`.
- `./gradlew nightly-test` — the cloud-object-store tags (`s3`, `google-cloud`, `azure`).

Each of those takes the same redirect: `./gradlew property-test > build/test-run.log 2>&1`.

## Module addressing

Modules are named `xtdb-<directory>`, matching the Maven artifact prefix.

- Top-level: `:xtdb-core:test`, `:xtdb-api:test`.
- Under `modules/`: `:modules:xtdb-kafka:test`, `:modules:xtdb-aws:test`.
- Bare `:test` is the root module, where most Clojure tests live (`src/test/clojure`).

## Test filtering

- Clojure namespaces use underscores in `--tests` patterns, not dashes — `xtdb.api_test`, not `xtdb.api-test`.
- `./gradlew :test --tests 'xtdb.api_test*'` — a namespace.
- `./gradlew :test --tests '*expression*'` — a wildcard.
- `./gradlew :test --tests '**can-manually-specify-system-time-47**'` — one test.
- Re-running the *same* `--tests` invocation is cached as UP-TO-DATE and does nothing.
  Add `--rerun-tasks` whenever the point of the run is to re-execute — verifying an intermittent failure, or checking a regenerated fixture.
  Add `--rerun-tasks` whenever the point of the run is to re-execute — verifying an intermittent failure, or checking a regenerated fixture.

## Iteration counts

There are **two** independent iteration knobs on `property-test`, and the obvious one only drives half the suite.

| Property | System property | Read by |
| --- | --- | --- |
| `-Piterations=N` | `xtdb.property-test-iterations` | `tu/property-test-iterations` in `src/testFixtures/clojure/xtdb/test_util.clj` — the `:num-tests` of the Clojure test.check properties |
| `-PsimulationIterations=N` | `xtdb.simulation-test-iterations` | `DEFAULT_ITERATIONS` in `core/src/test/kotlin/xtdb/SimulationTestBase.kt` — the invocation count of each `@RepeatableSimulationTest` |

Both default to 100.

So `./gradlew property-test --tests '*SimulationTest*' -Piterations=500` runs **100** iterations per simulation method, not 500 — you get a second fresh-seed run of the same length, not a longer one.
This has nothing to do with `--tests`; `-Piterations` simply does not reach the Kotlin simulations.
To lengthen a simulation run, pass `-PsimulationIterations=N`.

You MUST read the actual iteration count out of the run's totals or the agent's report before claiming a higher-iteration run was performed.

Two per-method overrides beat both properties:

- `@RepeatableSimulationTest(iterations = N)` fixes the count for that method.
- `@WithSeed(seed = N)` pins the seed and runs exactly one iteration — this is how you reproduce a reported failure, since `SeedExtension` logs `Test failed with seed: …` and reraises as `AssertionError("Test threw an exception (seed=…)")`.

## Simulation tests are invisible to `./gradlew test`

The seeded simulation classes carry `@Tag("property")` at class level, and `./gradlew test` excludes that tag.
A change to **indexing, compaction or GC** that breaks them therefore looks green locally and only fails in CI's property job.

If you have touched those subsystems you MUST also run `./gradlew property-test`.

The classes concerned:

- `core/src/test/kotlin/xtdb/NodeSimulationTest.kt`
- `core/src/test/kotlin/xtdb/cache/CacheSimulationTest.kt`
- `core/src/test/kotlin/xtdb/compactor/CompactorSimulationTest.kt`
- `core/src/test/kotlin/xtdb/indexer/LogProcessorSimTest.kt`
- `modules/postgres-source/src/test/kotlin/xtdb/postgres/PostgresSourceSimulationTest.kt`
- `modules/postgres-source/src/test/kotlin/xtdb/postgres/PostgresSourceTypesPropertyTest.kt`

## Typical durations

Budget for compilation and reporting overhead as well as the tests themselves.

- Single namespace: 30–60s.
- Module test suite: 2–5 min.
- Full project suite: 10+ min.
- Integration tests: 5–15 min, I/O bound.
- Property tests: varies with iteration count.

## What a run does not prove

- **`:compileTestClojure` is the cheap Clojure check; `:compileClojure` is not.**
  The root project has no `src/main/clojure` at all, so `:compileClojure` has nothing to compile and goes green without loading a line.
  `:compileTestClojure` and `:compileTestFixturesClojure` AOT-compile every test and fixture namespace — no `--tests` filter applies — which loads everything those namespaces require, module Clojure included, so a namespace that no longer loads fails there.
  Module `main` Clojure is not AOT-compiled and nothing checks it in its own right; a test namespace requiring it is what catches it.
- **A green run does not prove absence of reflection or boxed math.**
  `test` never sets `*warn-on-reflection*`, and `*unchecked-math*` reaches only the files that `set!` it themselves.
  `./gradlew codegen-report` (~90s, reports and always exits 0) covers the code the expression engine generates at runtime — the per-row path, which no load-time check can see, because a template compiles cleanly whatever the form it emits compiles to.
  **Discard its stdout and read the file** — the console copy is preceded by several thousand lines of node logs, and `build/codegen-report.txt` is the whole report including any namespace that failed to load or died part-way:
  ```
  ./gradlew -q codegen-report > /dev/null 2>&1; cat build/codegen-report.txt
  ```
  **Its coverage is the coverage of the tests it runs**: silence about an emitter branch you have just added means no test reached it, not that it is clean. Exercising the new branch is a precondition of checking it, not a follow-up.
  For ordinary Clojure outside the EE, `./gradlew reflection-check -Pns=xtdb.pgwire` compiles that namespace from source with the flag on and prints file and line. It warns about the whole transitive cone, so grep for your own file.
  Between the two, don't add a type hint speculatively on a reviewer's say-so — check it.

## When a test fails

**All tests pass on `main`. There are no pre-existing failures.**
If a test fails after your change, you broke it.
Investigate your own diff, find the bug, fix it.

- You MUST NOT speculate that a failure might be pre-existing.
- You MUST NOT stash your changes or check out `main` to "verify" that theory.
- You MUST NOT disable, skip or loosen an assertion to get to green.

The one carve-out, and it is deliberately narrow:

- You MAY check `gh issue list --label flaky` **if you genuinely believe the failure is a known flake** — the failing test is in a subsystem your change does not touch, or the failure message is about timing, ordering or resource contention rather than about behaviour.
- An open issue labelled `flaky` that matches the failure you are looking at is the **only** acceptable evidence.
  "It passed on the retry", "it looks racy", and "this test is known to be slow" are not.
- Absent a matching issue, the failure is yours. Fix it.
- If the failure is a genuine flake with no issue yet, open one and label it `flaky` rather than passing the problem on silently.

## Regenerating arrow-edn golden fixtures

Several namespaces assert a live run against committed `.arrow.edn` fixtures under `src/test/resources/xtdb/` — `xtdb.log-test`, `xtdb.indexer-test`, `xtdb.indexer.live-index-test`, `xtdb.indexer.live-table-test`, `xtdb.database-test`, `xtdb.metadata-test`, `xtdb.compactor-test`.
`xtdb.check-pbuf` reads the same toggle for its `.binpb.edn` fixtures, so anything below applies to those too.

To regenerate after an intended serialization change:

1. Uncomment the `#_aet/wrap-regen` line in that namespace's `use-fixtures` — it binds `xtdb.arrow-edn-test/*regen?*` true for that namespace only.
   Prefer this over flipping the `*regen?*` default: a blanket regen rewrites every fixture the run touches and masks unintended drift.
2. Run the namespace with `--rerun-tasks` — Gradle caches a repeated `--tests` invocation as UP-TO-DATE and does nothing.
3. Copy the regenerated files back into `src/test/resources/`, then re-comment the toggle and re-run with `--rerun-tasks` to verify green.

Gotchas, all of which have cost real time:

- **The output does not land in `src/`.**
  Expected paths resolve through `io/resource`, which under the Gradle `test` task is `build/resources/test/xtdb/…`.
  `git status src/` after a regen run shows nothing; you MUST copy the files back yourself.
- **A regen run tells you nothing about correctness.**
  `check-arrow-edn-dir` writes the expected file from the actual one and *then* compares, so it is trivially green; `maybe-write-arrow-edn!` (the `xtdb.log-test` shape) reads the old fixture before writing the new one, so that assertion fails exactly once by design and the new bytes land anyway.
  Either way the only meaningful verification is a re-run with the toggle off.
- **Do NOT `git rm` a fixture to force a regen.**
  `io/resource` returns nil for a missing resource and the write path breaks — regen only works against a fixture that already exists.
- **Comparison walks the expected tree**, so a file the run produces that the fixture lacks is silently unchecked.
  A genuinely new fixture file has to arrive via the regen path.
- **Both toggles are tagged `<<no-commit>>`** and the `.githooks/pre-commit` hook aborts a commit whose staged diff contains that marker.
  If the hook fires, you left a toggle on — don't `--no-verify` past it.
- **Never pin `xt$txs` in a golden file.**
  A tx-id is a message id derived from the log offset, so the same sequence of transactions yields different tx-ids from run to run.