Adding Tests and CI
Why this is a cost question
Every agent-native workflow draws on one org-wide pool of GitHub-hosted runners.
On the Team plan it held 60 concurrent jobs, and on 2026-09-29 the Fast tests
gate waited a median 445 s just for a slot. BuilderIO moved to Enterprise
Cloud in October 2026, which raises the pool to 500 jobs (50 macOS). Minutes
are free for this OSS repo; slots are shared with every private repo. Scope
still matters, because a job a change cannot affect delays other PRs once the
pool fills, but parallelism that cuts the critical path is now worth a slot.
scripts/ci-change-scope.ts is the one place that decides what a change runs.
It classifies the changed paths into a full or targeted run and emits one output
per check. ci.yml jobs gate on those outputs. A .github/** change, the
lockfile, root config, or a scripts/** change other than a guard script or
root script test forces a full run.
Adding a test
First decide whether the test should exist: test-audit owns that gate and
its junk patterns. This section covers where it lives and what runs it.
- Put it in the workspace of the code it proves. Targeted runs test the
changed packages and their dependents (
...{packages/core}), so a test
in another package either runs on unrelated changes or never runs on
the one that breaks it.
- Prove it at the cheapest tier that can fail for the bug: a pure
function, then a component or
.db.test.ts, then browser E2E last.
Some E2E suites, such as Design's, run only after merge and cannot fail
a PR. Do not restate one assertion at several tiers.
design-editor-architecture covers failing-first proof and mutation
audits.
- A new test joins an existing lane or job. It never gets a job of its
own. A changed root
scripts/*.test.ts, or the test beside a changed
guard script, runs in Security guards through the script_tests
output.
- Mind what
vitest --changed cannot see. Targeted Core lanes run only
the tests whose import graph reaches a changed file. A test that reads a
file from disk (a skill, a fixture, a template) will not rerun when only
that file changes. Import it, or teach requiresFullCoreFastTests in
scripts/ci-test-lanes.ts about the path.
- Prefer an oracle to hand-picked expected values. Sometimes a reference
decides the right answer: a public spec through a library (RFC 5545 via
rrule), the schema, or an older implementation. Then write a fast-check
property against it. templates/calendar/server/lib/series-recurrence.oracle.spec.ts
found three recurrence bugs that the example tests had passed over. Mark a
confirmed bug it.fails with an // oracle finding: comment. Never loosen
the property instead.
- A surviving mutant is a line no test can fail on.
pnpm mutation:changed
runs Stryker on the lines this branch added in packages/core/src. CI runs it
as Mutation test changed Core lines.
- Snapshot output is for reading; reviewers read its diff beside the code diff.
- Never skip, disable, or quarantine a test to get green.
guard:no-oracle-loosening fails a branch that adds .skip/.only, test
retries, or passWithNoTests, or that removes more assertions from a test
file than it adds. The exception is when an oracle-change: <reason> comment
says what behaviour changed.
Writing a browser E2E test
A browser test is the most expensive tier, so it has to fail when the feature
breaks the way a user would see it. A test that only checks that something
happened stays green while the result is wrong.
- Assert the exact end state, stated literally: the saved parent, order,
position, and size. A fired message,
toContain, a file over N bytes, or a
loose closeTo passes over the bug.
- Read persisted state, not only the DOM. Wait for the save to settle, then
read what the server stored. An edit that never saved must fail.
- Fail on collateral change. Diff the saved document before and after, and
allow only the nodes the step meant to touch.
- Fail on app errors by default, in a fixture: console and page errors,
app responses of 400 or more, error toasts, a save still in flight at
teardown, and page loads the test did not start. A request that fails and is
then retried still counts.
- Own the data. Create a fresh record per test and delete it after the
page leaves. Never mutate a shared seed or delete a record under an open
editor.
- Drive the real input path at any zoom. Click, drag, and type at positions
computed from the app's own geometry; constant page-pixel offsets break when
the default zoom changes.
- One flow per test, a guard before each step, no conditionals, no sleeps.
In a serial chain, one failure skips everything after it.
- Keep artifacts out of the watched source tree. A file written there
reloads the dev server mid-test; use
testInfo.outputPath.
- Check intent before calling it a bug. Read the source first; some saved
attributes that look like leaks are deliberate metadata. A test that asserts
correct behaviour and fails is a bug report: never loosen it to pass.
Design's harness (templates/design/e2e/rails.ts) and its traps are in
references/design-e2e.md.
Test workers
Fast-test lanes run Vitest with VITEST_CONCURRENCY=100%, one worker per
core. The shared config's 25% default is for laptops, and on CI's 4-vCPU
runners it meant one worker: a core shard took 542 s at one worker and 201 s
at four, on the same CPU time. On the 60-slot pool that let two full-width
lanes replace five, but each lane then ran ~18 min of packages one after
another. With 500 slots CI plans eight lanes (LANES in ci.yml): each costs
~100 s of setup, and the planner balances lanes by test-file count.
Large packages are sharded too. splitLargePackages in
scripts/ci-test-lanes.ts splits a package heavier than a fair lane share
into Vitest --shards, so Design no longer sets the floor for every lane.
Only a bare vitest test script can be sharded, and no shard drops below
MIN_SHARD_FILES.
Let CI override the worker count. A package that pins maxWorkers
goes through resolveMaxWorkers(process.env, fallback); a literal
overrides the CI setting.
Leave headroom for work inside a test body. Booting PGlite or
importing a server bundle takes several times longer when every core is
busy, and a different test crosses the limit each run. The shared config
allows 30 s; do not add a stricter per-block timeout.
Give parallel workers separate databases. A file that opens the
database without DATABASE_URL gets the default directory, whose lock
admits one process at a time. Core's Vitest setup gives each worker slot
its own under a root that belongs to one run, so concurrent runs never
share one; a test that asserts the default URL clears DATABASE_URL
itself.
Keep the forks pool with isolation. Threads ran slower and failed 79
tests; turning isolation off gained 7% and failed 129.
Adding or changing a CI job
| Rule |
Why, from this repo |
Give a job its own check in CHECK_NAMES whose condition names the paths that can change its result. Never borrow another job's output. |
The Postgres connection budget reused neon_query_budget and ran on every template edit. Its probe imports only @agent-native/core/db. |
| Per-app work takes a list output, not a boolean. Shared packages or a full run select every app; a template change selects that template; an empty selection turns the check off. If the check is on and the list is empty, the job fails. |
query_budget_apps and ssr_boot_apps: a one-template PR built and measured all 16 templates (12–13 min). When shared code selects every template, query_budget_matrix splits them across two jobs. The beta publisher's discover-sites publishes only sites whose dependency closure changed. |
| Split the expensive step from the cheap one. Keep the cheap check on every run and gate the expensive step on its narrower inputs. |
Android: expo export runs whenever a file Metro bundles changes, but the Gradle compile, the bulk of the ~23 min Android job, runs only when packages/mobile-app, the lockfile, or the workflow changed. |
A path filter covers the job's whole dependency closure, including install-time inputs: root package.json, pnpm-workspace.yaml, the prebuild script, and the lockfile. |
The desktop canary filter missed the bundled Chrome extension and then the postinstall inputs. Each gap skipped runs that should have caught a break. guard:mobile-build-paths traces Metro's imports and fails when the mobile filter misses one. |
Subscribe only to events that can change the outcome. Validators pin some trigger sets, e.g. scripts/validate-*-workflow.ts and scripts/package-release-workflow.test.ts; update them in the same change. |
Content product conformance reran on labeled/unlabeled but never reads labels, and on title edits although it reads the body only for its declaration. |
Do not add a job for under a minute of work. Checkout, install, and a pool slot cost more than the check. Fold it into a job with the same setup, and do not duplicate what pnpm guards already runs. |
Consolidation took PR pushes from ~31 to ~23 jobs. It folded PGlite locking into Content DB tests, privacy evals into Brain evals, QA static into Security guards, and the changeset check into Lint & format, and dropped a drizzle guard job pnpm guards covered. |
| Batch periodic publishing on a schedule with change detection, instead of once per merge. |
Nightly npm snapshots publish every 3 h, and only when a publishable path changed since the last successful scheduled run. |
PR workflows cancel superseded runs: group: <name>-${{ github.event.pull_request.number || github.ref }} with cancel-in-progress: true. Publishers on main use cancel-in-progress: false so a deploy is never killed mid-flight. An event rejected only by a job if still joins the run's group and cancels a real run, so filter it in on: or give ignored events a throwaway group. |
Visual Recap events the gate ignored used to cancel an in-progress recap; they now get a throwaway group. |
| When scope cannot be computed, run everything or fail. Never skip. |
An unreadable beta site list, an unknown base, or unrelated history publishes every site. A failed git diff fails the step; it is never || true. |
| Print what fails, not everything. |
The full-tree lint printed ~15k pre-existing warnings over the one error. It now runs oxlint --quiet. |
A skipped job counts as passing for required checks, so gating a required job
or step with if is safe. Every uppercase key a workflow adds under env:
needs a docs/environment-variables.md row (guard:env-documentation). Prefer
built-in GITHUB_* variables over new ones.
Proving a CI change
- Classifier: add
scripts/ci-change-scope.test.ts cases for each new
check: on, off, full run, and tooling-only full run. Then run
node --experimental-strip-types --test scripts/ci-change-scope.test.ts scripts/ci-build-workspaces.test.ts scripts/ci-test-lanes.test.ts.
- Real history: run
scripts/ci-change-scope.ts with GITHUB_OUTPUT set
on real main commit ranges, and put the before/after in the PR.
- Shell: run each new
run: snippet locally against a matching input
and an empty one. Run actionlint on the workflow when it is available.
- Mind the self-test gap: a PR that edits
ci.yml runs full CI itself,
so its own checks cannot show the saving. The classifier output is the
evidence.
Real failures this replaces
- "whats up with Cold-request query budget. runs on every CI for every
template? seems wrong"
- "fast lane tests seem absuredly expensive"
- "why is Generate + run standalone Chat running so frequently? shouldnt it
listen to Chat changes?"
- "so tldr: ~2-3 simultaneous PRs will eat up our entire quota of jobs right"
- "can you make lint & format not print all the warnings?"
1---2name: adding-tests-and-ci3description: >-4 Where a new test belongs, how to write a browser E2E test that fails when the5 feature breaks, and how to scope CI jobs, steps, and workflow triggers so6 they run only when a change can affect them. Use when adding, moving, or7 writing a test (including Design E2E specs), adding or changing a CI job,8 step, or workflow, or when a job is slow or runs on unrelated changes.9scope: dev10metadata:11 source: https://skillmd.com/skills/builderio/adding-tests-and-ci12 internal: true13---1415# Adding Tests and CI1617## Why this is a cost question1819Every agent-native workflow draws on one org-wide pool of GitHub-hosted runners.20On the Team plan it held 60 concurrent jobs, and on 2026-09-29 the `Fast tests`21gate waited a median 445 s just for a slot. BuilderIO moved to Enterprise22Cloud in October 2026, which raises the pool to 500 jobs (50 macOS). Minutes23are free for this OSS repo; slots are shared with every private repo. Scope24still matters, because a job a change cannot affect delays other PRs once the25pool fills, but parallelism that cuts the critical path is now worth a slot.2627`scripts/ci-change-scope.ts` is the one place that decides what a change runs.28It classifies the changed paths into a full or targeted run and emits one output29per check. `ci.yml` jobs gate on those outputs. A `.github/**` change, the30lockfile, root config, or a `scripts/**` change other than a guard script or31root script test forces a full run.3233## Adding a test3435First decide whether the test should exist: `test-audit` owns that gate and36its junk patterns. This section covers where it lives and what runs it.3738- **Put it in the workspace of the code it proves.** Targeted runs test the39 changed packages and their dependents (`...{packages/core}`), so a test40 in another package either runs on unrelated changes or never runs on41 the one that breaks it.42- **Prove it at the cheapest tier that can fail for the bug:** a pure43 function, then a component or `.db.test.ts`, then browser E2E last.44 Some E2E suites, such as Design's, run only after merge and cannot fail45 a PR. Do not restate one assertion at several tiers.46 `design-editor-architecture` covers failing-first proof and mutation47 audits.48- **A new test joins an existing lane or job.** It never gets a job of its49 own. A changed root `scripts/*.test.ts`, or the test beside a changed50 guard script, runs in `Security guards` through the `script_tests`51 output.52- **Mind what `vitest --changed` cannot see.** Targeted Core lanes run only53 the tests whose import graph reaches a changed file. A test that reads a54 file from disk (a skill, a fixture, a template) will not rerun when only55 that file changes. Import it, or teach `requiresFullCoreFastTests` in56 `scripts/ci-test-lanes.ts` about the path.57- **Prefer an oracle to hand-picked expected values.** Sometimes a reference58 decides the right answer: a public spec through a library (RFC 5545 via59 `rrule`), the schema, or an older implementation. Then write a fast-check60 property against it. `templates/calendar/server/lib/series-recurrence.oracle.spec.ts`61 found three recurrence bugs that the example tests had passed over. Mark a62 confirmed bug `it.fails` with an `// oracle finding:` comment. Never loosen63 the property instead.64- **A surviving mutant is a line no test can fail on.** `pnpm mutation:changed`65 runs Stryker on the lines this branch added in `packages/core/src`. CI runs it66 as `Mutation test changed Core lines`.67- Snapshot output is for reading; reviewers read its diff beside the code diff.68- Never skip, disable, or quarantine a test to get green.69 `guard:no-oracle-loosening` fails a branch that adds `.skip`/`.only`, test70 retries, or `passWithNoTests`, or that removes more assertions from a test71 file than it adds. The exception is when an `oracle-change: <reason>` comment72 says what behaviour changed.7374## Writing a browser E2E test7576A browser test is the most expensive tier, so it has to fail when the feature77breaks the way a user would see it. A test that only checks that something78happened stays green while the result is wrong.7980- **Assert the exact end state, stated literally:** the saved parent, order,81 position, and size. A fired message, `toContain`, a file over N bytes, or a82 loose `closeTo` passes over the bug.83- **Read persisted state, not only the DOM.** Wait for the save to settle, then84 read what the server stored. An edit that never saved must fail.85- **Fail on collateral change.** Diff the saved document before and after, and86 allow only the nodes the step meant to touch.87- **Fail on app errors by default, in a fixture:** console and page errors,88 app responses of 400 or more, error toasts, a save still in flight at89 teardown, and page loads the test did not start. A request that fails and is90 then retried still counts.91- **Own the data.** Create a fresh record per test and delete it after the92 page leaves. Never mutate a shared seed or delete a record under an open93 editor.94- **Drive the real input path at any zoom.** Click, drag, and type at positions95 computed from the app's own geometry; constant page-pixel offsets break when96 the default zoom changes.97- **One flow per test, a guard before each step, no conditionals, no sleeps.**98 In a serial chain, one failure skips everything after it.99- **Keep artifacts out of the watched source tree.** A file written there100 reloads the dev server mid-test; use `testInfo.outputPath`.101- **Check intent before calling it a bug.** Read the source first; some saved102 attributes that look like leaks are deliberate metadata. A test that asserts103 correct behaviour and fails is a bug report: never loosen it to pass.104105Design's harness (`templates/design/e2e/rails.ts`) and its traps are in106[references/design-e2e.md](references/design-e2e.md).107108## Test workers109110Fast-test lanes run Vitest with `VITEST_CONCURRENCY=100%`, one worker per111core. The shared config's 25% default is for laptops, and on CI's 4-vCPU112runners it meant one worker: a core shard took 542 s at one worker and 201 s113at four, on the same CPU time. On the 60-slot pool that let two full-width114lanes replace five, but each lane then ran ~18 min of packages one after115another. With 500 slots CI plans eight lanes (`LANES` in `ci.yml`): each costs116~100 s of setup, and the planner balances lanes by test-file count.117118- **Large packages are sharded too.** `splitLargePackages` in119 `scripts/ci-test-lanes.ts` splits a package heavier than a fair lane share120 into Vitest `--shard`s, so Design no longer sets the floor for every lane.121 Only a bare `vitest` test script can be sharded, and no shard drops below122 `MIN_SHARD_FILES`.123124- **Let CI override the worker count.** A package that pins `maxWorkers`125 goes through `resolveMaxWorkers(process.env, fallback)`; a literal126 overrides the CI setting.127- **Leave headroom for work inside a test body.** Booting PGlite or128 importing a server bundle takes several times longer when every core is129 busy, and a different test crosses the limit each run. The shared config130 allows 30 s; do not add a stricter per-block timeout.131- **Give parallel workers separate databases.** A file that opens the132 database without `DATABASE_URL` gets the default directory, whose lock133 admits one process at a time. Core's Vitest setup gives each worker slot134 its own under a root that belongs to one run, so concurrent runs never135 share one; a test that asserts the default URL clears `DATABASE_URL`136 itself.137- **Keep the forks pool with isolation.** Threads ran slower and failed 79138 tests; turning isolation off gained 7% and failed 129.139140## Adding or changing a CI job141142| Rule | Why, from this repo |143| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |144| Give a job its own check in `CHECK_NAMES` whose condition names the paths that can change its result. Never borrow another job's output. | The Postgres connection budget reused `neon_query_budget` and ran on every template edit. Its probe imports only `@agent-native/core/db`. |145| Per-app work takes a list output, not a boolean. Shared packages or a full run select every app; a template change selects that template; an empty selection turns the check off. If the check is on and the list is empty, the job fails. | `query_budget_apps` and `ssr_boot_apps`: a one-template PR built and measured all 16 templates (12–13 min). When shared code selects every template, `query_budget_matrix` splits them across two jobs. The beta publisher's `discover-sites` publishes only sites whose dependency closure changed. |146| Split the expensive step from the cheap one. Keep the cheap check on every run and gate the expensive step on its narrower inputs. | Android: `expo export` runs whenever a file Metro bundles changes, but the Gradle compile, the bulk of the ~23 min Android job, runs only when `packages/mobile-app`, the lockfile, or the workflow changed. |147| A path filter covers the job's whole dependency closure, including install-time inputs: root `package.json`, `pnpm-workspace.yaml`, the prebuild script, and the lockfile. | The desktop canary filter missed the bundled Chrome extension and then the postinstall inputs. Each gap skipped runs that should have caught a break. `guard:mobile-build-paths` traces Metro's imports and fails when the mobile filter misses one. |148| Subscribe only to events that can change the outcome. Validators pin some trigger sets, e.g. `scripts/validate-*-workflow.ts` and `scripts/package-release-workflow.test.ts`; update them in the same change. | Content product conformance reran on `labeled`/`unlabeled` but never reads labels, and on title edits although it reads the body only for its declaration. |149| Do not add a job for under a minute of work. Checkout, install, and a pool slot cost more than the check. Fold it into a job with the same setup, and do not duplicate what `pnpm guards` already runs. | Consolidation took PR pushes from ~31 to ~23 jobs. It folded PGlite locking into Content DB tests, privacy evals into Brain evals, QA static into Security guards, and the changeset check into Lint & format, and dropped a drizzle guard job `pnpm guards` covered. |150| Batch periodic publishing on a schedule with change detection, instead of once per merge. | Nightly npm snapshots publish every 3 h, and only when a publishable path changed since the last successful scheduled run. |151| PR workflows cancel superseded runs: `group: <name>-${{ github.event.pull_request.number \|\| github.ref }}` with `cancel-in-progress: true`. Publishers on `main` use `cancel-in-progress: false` so a deploy is never killed mid-flight. An event rejected only by a job `if` still joins the run's group and cancels a real run, so filter it in `on:` or give ignored events a throwaway group. | Visual Recap events the gate ignored used to cancel an in-progress recap; they now get a throwaway group. |152| When scope cannot be computed, run everything or fail. Never skip. | An unreadable beta site list, an unknown base, or unrelated history publishes every site. A failed `git diff` fails the step; it is never `\|\| true`. |153| Print what fails, not everything. | The full-tree lint printed ~15k pre-existing warnings over the one error. It now runs `oxlint --quiet`. |154155A skipped job counts as passing for required checks, so gating a required job156or step with `if` is safe. Every uppercase key a workflow adds under `env:`157needs a `docs/environment-variables.md` row (`guard:env-documentation`). Prefer158built-in `GITHUB_*` variables over new ones.159160## Proving a CI change161162- **Classifier:** add `scripts/ci-change-scope.test.ts` cases for each new163 check: on, off, full run, and tooling-only full run. Then run164 `node --experimental-strip-types --test scripts/ci-change-scope.test.ts scripts/ci-build-workspaces.test.ts scripts/ci-test-lanes.test.ts`.165- **Real history:** run `scripts/ci-change-scope.ts` with `GITHUB_OUTPUT` set166 on real `main` commit ranges, and put the before/after in the PR.167- **Shell:** run each new `run:` snippet locally against a matching input168 and an empty one. Run `actionlint` on the workflow when it is available.169- **Mind the self-test gap:** a PR that edits `ci.yml` runs full CI itself,170 so its own checks cannot show the saving. The classifier output is the171 evidence.172173## Real failures this replaces174175- "whats up with Cold-request query budget. runs on every CI for every176 template? seems wrong"177- "fast lane tests seem absuredly expensive"178- "why is Generate + run standalone Chat running so frequently? shouldnt it179 listen to Chat changes?"180- "so tldr: ~2-3 simultaneous PRs will eat up our entire quota of jobs right"181- "can you make lint & format not print all the warnings?"