Writing E2E flows
Overview
A flow is one user goal, driven in a real browser against the whole running system and asserted in
both the UI and the backend state behind it. This skill takes you from "the feature I'm working on"
to a flow that is green on a managed stack and has been shown to fail when the thing it claims to
prove is broken — because an assertion nobody has watched fail is not coverage, it is decoration.
When to use / when not to use
Use it when a change adds or alters behaviour a user can see; when a diff moves an endpoint, route,
table or selector some flow has pinned; when a review says E2E coverage is missing; or when someone
asks how to test a FutureAGI behaviour end to end.
Do not use it for frontend unit tests (frontend/src/**/__tests__), backend pytest
(futureagi/**/tests/), the browserless API-journey scripts (frontend/scripts/api-journeys/), or
for changing the harness itself. Extending e2e/lib/ is its own piece of work — see step 0.5.
Read this first (the contract)
Read all three before you plan anything. They are the contract; this skill only adds what they do
not carry — the decision procedure, the interview, design-doc mining, the pinned-facts inventory,
the harness gaps, and the proof loop.
e2e/README.md § "Writing a flow" — in full, every time. Not skimmed, not remembered.e2e/FLOWS.md — what already exists, which area owns it, and the next free id.- The newest spec in the target area; if the area has no flows yet,
e2e/flows/observe/trace-ingestion.spec.ts, the canonical exemplar the README reproduces.
Procedure
Track the numbered steps as todos, one each.
0. Ground (read first, ask second)
Read the three documents above.
Classify the change. From the repo root, with the branch checked out:
node e2e/scripts/e2e-coverage.mjs --pr <n> --json # when a PR exists
node e2e/scripts/e2e-coverage.mjs --base origin/<base> --title "<branch or PR title>" --json
Always pass --title in the no-PR form. One of the new-surface signals is a feat title in an
area with no flows, and without a title that signal cannot fire — the same diff then classifies
weaker than it will once the PR exists. --pr reads the title from GitHub.
Prefer --pr; it takes the base from GitHub. Do not pass --base alongside it — an explicit
--base wins over the PR's real base (e2e/scripts/e2e-coverage.mjs:resolveBase), and
hand-resolving is precisely how a stacked branch gets classified against the wrong tree. With no
PR, take the base from the branch's upstream or merge-base; never assume dev.
It returns classification (NEW-FLOW / UPDATE-EXISTING / EXEMPT / UNDETERMINED), areas,
areasWithoutFlows, hits (flows whose pinned surface the diff touches), newSurfaceSignals,
marker, verdict and explanation. Two things to know: --pr reads the PR's title, body and
base from GitHub but classifies the local checkout's diff, so the branch has to be checked
out first; and an empty diff comes back EXEMPT (no-changes), which means "nothing to classify",
not "no flow needed".
If the script is not in the checkout: read FLOWS.md and classify by hand — does any listed
flow's area, endpoint or route appear in your diff (update it), or is this user-visible surface
no flow covers (new flow)?
Resolve the ticket. If a Linear integration is available, read the issue — title, description,
## Acceptance Criteria, ## Scope, and the parent epic. If not, ask the user to paste the
acceptance criteria or point you at the doc. Do not infer the acceptance criteria from the diff:
the diff says what changed, the ticket says what a user is now supposed to be able to do.
If the ticket or branch names a feature, find its design doc with the procedure in
references/design-doc-mining.md — read it when you have a
ticket id, a feature name, or a feat(<scope>) PR title and no doc in hand. Read only the
sections that map to a lane: PRD requirements and success criteria feed the steps; UI-UX
interaction flows and empty/error states feed the UI lane; the API contract feeds the API lane;
lifecycle, edge cases and data stores feed the storage lane and the poll budget; a TEST_PLAN.md
§ E2E scenarios, where one exists, is a ready-made step list.
Check the flow you are about to plan against
references/harness-gaps.md — read it before every plan. A flow that
needs a second user, a per-test empty org, a file upload, a custom OTLP shape, a programmable LLM
answer, an email sink, or a service the stack does not start is blocked on a harness
extension. That extension is its own piece of work, planned with the user, never assembled
inside a spec.
1. Interview — ask only when one of these is true
One question per message, multiple choice where the options are known, and say which classification
step 0 produced before you ask.
- More than one user goal is plausible for the change.
- The behaviour may not be reachable on the OSS image or this stack profile — EE-only,
serving,
code-executor, the simulation runner, or a CDC table whose mirror is known-drifted.
- Which backend state proves the goal is not derivable from the design doc or the serializer.
- A locator would need a new
data-testid — that is, after grepping product code for the hooks
that already exist (grep -rn 'data-[a-z-]*=' frontend/src/sections frontend/src/components;
section directories are not named after e2e areas, so grep the whole tree) nothing fits. Say so: a
new one ships as its own small product PR first, per the README, and until it is released the flow
cannot run on the managed stack.
- Step 0.5 hit a harness gap.
Nothing else is worth interrupting for. If none of these is true, go straight to the plan.
2. Flow plan — HARD GATE
Present the plan in the shape of
references/flow-plan-template.md — read that file before writing
the plan — and stop.
Do not create or edit any file under e2e/ until the plan is approved by whoever is directing
this session — a person, or the agent orchestrating it. Not a skeleton, not "just the
annotation", not a draft in a scratch directory to be moved later. A plan nobody has answered is
not an approved plan, and neither is silence. An approval that comes back from the party who gave
you this work is the approval; do not stall for a second one because it did not arrive from a
human.
The plan names: id and area; userGoal; steps in order; backendChecks; for each check its
lane, its poll budget named by what it waits on, and the test.setTimeout sum when budgets chain
past 120 s; the seeding strategy; the minted names and which assertion anchors on which; and the
tags.
3. Write
Follow the exemplar's structure in order: imports → pinned constants, each with a comment saying
where it was pinned from → test('<ID>: <sentence>', { tag, annotation: flowAnnotation({…}) }, fn)
→ test.setTimeout(...) as the first statement → seed → storage lane → UI lane → API lane →
dispose.
Every pinned constant carries its provenance in a comment: the frontend file, the serializer, or
"captured with page.on('request') from the running app". Wire bodies come from a captured request
or the DRF serializer, never from guessing — the create forms' resolvers rename fields, so the form's
shape is not the wire's shape.
Attach the seeded ids with testInfo.attach as you write the seeding step. CI uploads the HTML
report and nothing else, so a failure has to be readable from that report alone — the ids you seeded
are what lets anyone re-query the same rows afterwards.
Then, from e2e/:
yarn typecheck && yarn catalog
yarn catalog regenerates FLOWS.md. It is generated; never hand-write it or hand-edit it. Commit
it in the same commit as the spec.
4. Prove — definition of done
A flow is done when the spec attaches its seeded ids with testInfo.attach, so a CI failure is
readable from the report alone, and all four of these have happened, in order.
Iterate in attach mode against a stack you already have:
E2E_APP_URL=… E2E_API_URL=… E2E_COLLECTOR_URL=… E2E_CH_URL=… E2E_PG_URL=… bin/e2e test flows/<area>/<name>.spec.ts. Attach mode is for iteration, not for verdicts.
Managed run. If this branch changes product code the flow depends on, build it into the
images first — bin/e2e build backend|frontend|collector — then bin/e2e up with the version
variables the build prints, and run the flow there. Managed mode otherwise runs published
:latest images, so a flow that needs your change would pass or fail for the wrong reason.
Fallibility proof, once per backendCheck. Perturb the anchor the check rests on, run the
flow, and read the failure — perturbing one side only. In flows/evals/eval-task.spec.ts the
verdict constant feeds both the judge template's instructions and the assertion, so editing
it moves expected and actual together and the run stays green. Edit the input alone:
# in the spec, temporarily — inside the template `instructions` string only:
# "explanation": "${verdict}-BROKEN saw {{output}}"
bin/e2e test flows/evals/eval-task.spec.ts
# → expect(received).toBe(expected)
# Expected: "e2e-verdict-w0-m1x2y3 saw llm"
# Received: "e2e-verdict-w0-m1x2y3-BROKEN saw llm"
Perturb the side under test, never a value shared by the assertion and the input — if the run
stays green, the perturbation proved nothing. Quote the failure in your notes, restore the file,
and run green again. A check that still passes perturbed is not a check.
Record the evidence: the exact commands, the last green run's summary, the quoted failure per
check, and the E2E: new <ID> (or updated <ID>) line for the PR body.
5. Hand off
Spec and FLOWS.md in the same commit. No test.only — CI runs forbidOnly. No retries: a flow
that passes on the second attempt is telling you something true. Never pass --grep-invert on the
command line; it replaces the config's own value and un-quarantines the whole suite. Quarantine only
with an owner, a ticket and an expiry within 45 days. The PR body carries the evidence from step 4.
State plainly which of step 4's runs you did not do, and why. A flow handed over without its managed
run or its fallibility proof is a draft; naming that is what stops a reviewer reading it as coverage.
Quick reference
Signatures, the pinned app facts (endpoints, page sizes, wire bodies, grid and filter selectors, the
in-container gateway address), the mock-LLM contract and the ClickHouse/Postgres binding traps are in
references/harness-cheatsheet.md — read it while writing the
spec.
| Primitive |
Import from |
Use it for |
test, expect |
../../lib/fixtures |
every flow that starts signed in |
test as base |
@playwright/test |
only a flow that drives login or signup itself |
actor fixture |
../../lib/fixtures |
worker-scoped org, keys, authed ApiClient |
probe fixture |
../../lib/fixtures |
ch / pg / apiList reads |
sendTrace |
../../lib/otlp |
seeding observe data through the collector |
POLL |
../../lib/state-probe |
the shared wait budgets below |
E2E |
../../lib/env |
endpoints — never hardcode a port |
flowAnnotation |
../../lib/flow-meta |
the catalog contract |
Pick a budget by what you are waiting on, never by which flow you are writing:
| Budget |
Waits on |
POLL.SPAN_VISIBLE (15 s) |
anything fi-collector writes — spans, curated traces |
POLL.EVAL_RESULT (90 s) |
a Temporal eval task reaching a terminal status |
POLL.CDC_VISIBLE (180 s) |
any CDC-fed table: tracer_eval_logger, model_hub_score, datasets, prompts, simulate, usage |
POLL.ASYNC_JOB (60 s) |
other Temporal work |
UI_READY (60 s, per-spec constant) |
every browser expect, toHaveURL, waitForResponse |
Red flags — STOP
| Thought |
Reality |
| "I'll add the annotation later." |
@flow without an annotation breaks yarn catalog for the whole suite; the catalog is CI's first gate, before the stack boots. |
"toContainText is fine here." |
It passes with the feature broken. The README's rule is exact sets: toHaveText([alpha]) proves the other rows are gone. |
| "I'll raise the project timeout." |
That slows every flow for one flow's budget. Raise this test's own ceiling with test.setTimeout, sized as the sum of the chained budgets. |
| "Skip the managed run, CI will catch it." |
CI builds the frontend from source, but a backend, collector or gateway your PR does not touch runs as the released :latest. A flow depending on an unreleased change to one of those is not exercised there at all. |
| "This spec has never been executed; the first run will confirm the locators." |
Then the first run is the author's, not the reviewer's. An unexecuted spec is a draft, and the plan you agreed to said "proven". |
| "It typechecks, it collects, the catalog renders — it's verified." |
All three pass on a flow whose every assertion is wrong. They check the shape, not the claim. Only step 4.3 checks the claim. |
| "Assert on the first row / on the count." |
actor is shared by every spec in the worker and nothing is ever deleted. Anchor on a value you minted; measure a delta where prior rows exist. |
| "The mock can just return X." |
It cannot be programmed. It echoes the last user message with fixed usage. Shape the prompt so the echo parses — see the cheatsheet's recipe. |
"Use 127.0.0.1." |
Login skips reCAPTCHA only when the request Host contains localhost. Address the stack as localhost. |
"I'll add the data-testid in this PR." |
Grep first — the hook you need usually exists. If it genuinely does not, the README still wants the attribute as its own small product PR, and your local managed run uses the released frontend until you bin/e2e build frontend. CI does build the frontend from source, so a same-PR attribute is exercised there. |
"There's no aria-pressed, so I'll assert the computed style — I check the unselected one too, so it can still fail." |
That answers a different objection. A locator bound to a CSS value breaks on any restyle and cannot be verified without the stack. It is the data-testid case; treat it as one. |
| "One retry won't hurt." |
retries: 0 is deliberate. A flow that passes on attempt two is reporting a real product behaviour; a retry throws that signal away. Quarantine with owner, ticket and expiry instead. |
"I'll read the spans table without FINAL." |
spans, traces and every CDC-fed table are ReplacingMergeTree. An unmerged read hands you the stale version of a row that moved through a lifecycle. |
| "This wait is a UI refresh, not an eventually-consistent store, so the default budget is right." |
The 10 s expect default is not a first-paint budget. A UI step that takes 10 s alone takes a minute beside other workers. Use the spec's UI_READY. |
"FLOWS.md is generated text; I'll hand-produce what the generator would emit." |
The gate is a byte compare, and the file says "do not edit" in its own header. Run yarn catalog. |
"login.spec.ts bypasses the fixtures, so I can assemble the actors I need by hand." |
That file is a documented single-actor exception because it drives the login UI. Multiple identities is a harness gap: propose the lib/ extension, do not build it inside a flow. |
| "The product's own path doesn't reach that state, so I'll drive the API instead." |
You have just changed what the flow proves. That is an interview question, not an implementation detail. |
| "The commit is mostly refactor; I know which claim is the user-visible one." |
Two agents reading the same commit picked two different goals. If more than one goal is plausible, ask. |
| "The diff tells me what changed; I don't need the ticket." |
The diff says what changed. The ticket says what a user can now do — which is what the flow asserts, and what its userGoal has to state. |
Common mistakes
Read references/footguns.md when a run fails for a reason you did not
predict — it is the full symptom → cause → fix list. The eight that bite first:
- Title is not exactly
<ID>: <sentence> — the catalog renders the id twice and quarantine by id
stops matching.
area in the annotation does not equal the flows/<area>/ directory — nothing validates it; it
silently opens a new section in FLOWS.md.- Forgetting
yarn catalog after touching a title, tag or annotation — CI fails before boot.
spans.trace_id bound as anything but {t:String}, or curated traces looked up by a
trace_id column that does not exist (it is keyed by id, bound {t:UUID}).- ClickHouse
count() compared as a number — it comes back a string; wrap it in Number(...).
probe.apiList used on an endpoint that does not return { result: { table } } — it hard-codes
that envelope.- In-container callers given a host URL — the eval judge runs in the worker container and must
address the gateway over the compose network.
yarn typecheck skipped — CI does not run it, and Playwright strips types without checking them.
Out of scope
- Retrofitting
data-testid across product code in this PR. One attribute, its own small PR, only
where the semantics are genuinely ambiguous.
- Extending
e2e/lib/ from inside a spec. A new fixture, probe or seeder is its own piece of work,
planned with the user.
- Flows for services this stack profile does not start —
serving, code-executor, the simulation
runner — or assertions on surfaces whose CDC mirror is known-drifted.
- Committing. Prepare the spec, the regenerated
FLOWS.md and the evidence; the user decides when
it lands.
1---2name: writing-e2e-flows3description: Use when a feature or fix in future-agi needs an end-to-end Playwright flow under e2e/ — a new flow for user-visible behaviour, an update to a flow whose pinned endpoint, route, table or selector changed, or when a review flagged missing E2E coverage. Also use to check whether an existing flow already pins an endpoint, route, table or selector you changed, and for 'how do I test this end to end' questions about the FutureAGI product. Not for frontend unit tests, backend pytest, the API-journey scripts, for running, operating, debugging or explaining the e2e stack and its harness scripts themselves — ports, compose profile, start-up time — or for the review-time verdict on whether a change needs a flow, which is the reviewing-prs coverage gate.4---56# Writing E2E flows78## Overview910A flow is one user goal, driven in a real browser against the whole running system and asserted in11both the UI and the backend state behind it. This skill takes you from "the feature I'm working on"12to a flow that is green on a managed stack **and** has been shown to fail when the thing it claims to13prove is broken — because an assertion nobody has watched fail is not coverage, it is decoration.1415## When to use / when not to use1617Use it when a change adds or alters behaviour a user can see; when a diff moves an endpoint, route,18table or selector some flow has pinned; when a review says E2E coverage is missing; or when someone19asks how to test a FutureAGI behaviour end to end.2021Do not use it for frontend unit tests (`frontend/src/**/__tests__`), backend pytest22(`futureagi/**/tests/`), the browserless API-journey scripts (`frontend/scripts/api-journeys/`), or23for changing the harness itself. Extending `e2e/lib/` is its own piece of work — see step 0.5.2425## Read this first (the contract)2627Read all three before you plan anything. They are the contract; this skill only adds what they do28not carry — the decision procedure, the interview, design-doc mining, the pinned-facts inventory,29the harness gaps, and the proof loop.3031- **`e2e/README.md` § "Writing a flow"** — in full, every time. Not skimmed, not remembered.32- **`e2e/FLOWS.md`** — what already exists, which area owns it, and the next free id.33- **The newest spec in the target area**; if the area has no flows yet,34 `e2e/flows/observe/trace-ingestion.spec.ts`, the canonical exemplar the README reproduces.3536## Procedure3738Track the numbered steps as todos, one each.3940### 0. Ground (read first, ask second)41421. Read the three documents above.43442. Classify the change. From the repo root, with the branch checked out:4546 ```47 node e2e/scripts/e2e-coverage.mjs --pr <n> --json # when a PR exists48 node e2e/scripts/e2e-coverage.mjs --base origin/<base> --title "<branch or PR title>" --json49 ```5051 Always pass `--title` in the no-PR form. One of the new-surface signals is a `feat` title in an52 area with no flows, and without a title that signal cannot fire — the same diff then classifies53 weaker than it will once the PR exists. `--pr` reads the title from GitHub.5455 Prefer `--pr`; it takes the base from GitHub. Do not pass `--base` alongside it — an explicit56 `--base` wins over the PR's real base (`e2e/scripts/e2e-coverage.mjs:resolveBase`), and57 hand-resolving is precisely how a stacked branch gets classified against the wrong tree. With no58 PR, take the base from the branch's upstream or merge-base; never assume `dev`.5960 It returns `classification` (NEW-FLOW / UPDATE-EXISTING / EXEMPT / UNDETERMINED), `areas`,61 `areasWithoutFlows`, `hits` (flows whose pinned surface the diff touches), `newSurfaceSignals`,62 `marker`, `verdict` and `explanation`. Two things to know: `--pr` reads the PR's title, body and63 base from GitHub but classifies **the local checkout's** diff, so the branch has to be checked64 out first; and an empty diff comes back `EXEMPT (no-changes)`, which means "nothing to classify",65 not "no flow needed".6667 If the script is not in the checkout: read `FLOWS.md` and classify by hand — does any listed68 flow's area, endpoint or route appear in your diff (update it), or is this user-visible surface69 no flow covers (new flow)?70713. Resolve the ticket. If a Linear integration is available, read the issue — title, description,72 `## Acceptance Criteria`, `## Scope`, and the parent epic. If not, ask the user to paste the73 acceptance criteria or point you at the doc. Do not infer the acceptance criteria from the diff:74 the diff says what changed, the ticket says what a user is now supposed to be able to do.75764. If the ticket or branch names a feature, find its design doc with the procedure in77 [references/design-doc-mining.md](references/design-doc-mining.md) — read it when you have a78 ticket id, a feature name, or a `feat(<scope>)` PR title and no doc in hand. Read only the79 sections that map to a lane: PRD requirements and success criteria feed the steps; UI-UX80 interaction flows and empty/error states feed the UI lane; the API contract feeds the API lane;81 lifecycle, edge cases and data stores feed the storage lane and the poll budget; a `TEST_PLAN.md`82 § E2E scenarios, where one exists, is a ready-made step list.83845. Check the flow you are about to plan against85 [references/harness-gaps.md](references/harness-gaps.md) — read it before every plan. A flow that86 needs a second user, a per-test empty org, a file upload, a custom OTLP shape, a programmable LLM87 answer, an email sink, or a service the stack does not start is **blocked on a harness88 extension**. That extension is its own piece of work, planned with the user, never assembled89 inside a spec.9091### 1. Interview — ask only when one of these is true9293One question per message, multiple choice where the options are known, and say which classification94step 0 produced before you ask.9596- More than one user goal is plausible for the change.97- The behaviour may not be reachable on the OSS image or this stack profile — EE-only, `serving`,98 `code-executor`, the simulation runner, or a CDC table whose mirror is known-drifted.99- Which backend state proves the goal is not derivable from the design doc or the serializer.100- A locator would need a **new** `data-testid` — that is, after grepping product code for the hooks101 that already exist (`grep -rn 'data-[a-z-]*=' frontend/src/sections frontend/src/components`;102 section directories are not named after e2e areas, so grep the whole tree) nothing fits. Say so: a103 new one ships as its own small product PR first, per the README, and until it is released the flow104 cannot run on the managed stack.105- Step 0.5 hit a harness gap.106107Nothing else is worth interrupting for. If none of these is true, go straight to the plan.108109### 2. Flow plan — HARD GATE110111Present the plan in the shape of112[references/flow-plan-template.md](references/flow-plan-template.md) — read that file before writing113the plan — and stop.114115**Do not create or edit any file under `e2e/` until the plan is approved by whoever is directing116this session — a person, or the agent orchestrating it.** Not a skeleton, not "just the117annotation", not a draft in a scratch directory to be moved later. A plan nobody has answered is118not an approved plan, and neither is silence. An approval that comes back from the party who gave119you this work _is_ the approval; do not stall for a second one because it did not arrive from a120human.121122The plan names: `id` and `area`; `userGoal`; `steps` in order; `backendChecks`; for each check its123lane, its poll budget named by what it waits on, and the `test.setTimeout` sum when budgets chain124past 120 s; the seeding strategy; the minted names and which assertion anchors on which; and the125tags.126127### 3. Write128129Follow the exemplar's structure in order: imports → pinned constants, each with a comment saying130where it was pinned from → `test('<ID>: <sentence>', { tag, annotation: flowAnnotation({…}) }, fn)`131→ `test.setTimeout(...)` as the first statement → seed → storage lane → UI lane → API lane →132`dispose`.133134Every pinned constant carries its provenance in a comment: the frontend file, the serializer, or135"captured with `page.on('request')` from the running app". Wire bodies come from a captured request136or the DRF serializer, never from guessing — the create forms' resolvers rename fields, so the form's137shape is not the wire's shape.138139Attach the seeded ids with `testInfo.attach` as you write the seeding step. CI uploads the HTML140report and nothing else, so a failure has to be readable from that report alone — the ids you seeded141are what lets anyone re-query the same rows afterwards.142143Then, from `e2e/`:144145```146yarn typecheck && yarn catalog147```148149`yarn catalog` regenerates `FLOWS.md`. It is generated; never hand-write it or hand-edit it. Commit150it in the same commit as the spec.151152### 4. Prove — definition of done153154A flow is done when the spec attaches its seeded ids with `testInfo.attach`, so a CI failure is155readable from the report alone, and all four of these have happened, in order.1561571. **Iterate in attach mode** against a stack you already have:158 `E2E_APP_URL=… E2E_API_URL=… E2E_COLLECTOR_URL=… E2E_CH_URL=… E2E_PG_URL=… bin/e2e test159flows/<area>/<name>.spec.ts`. Attach mode is for iteration, not for verdicts.1601612. **Managed run.** If this branch changes product code the flow depends on, build it into the162 images first — `bin/e2e build backend|frontend|collector` — then `bin/e2e up` with the version163 variables the build prints, and run the flow there. Managed mode otherwise runs published164 `:latest` images, so a flow that needs your change would pass or fail for the wrong reason.1651663. **Fallibility proof, once per `backendCheck`.** Perturb the anchor the check rests on, run the167 flow, and read the failure — perturbing **one side only**. In `flows/evals/eval-task.spec.ts` the168 `verdict` constant feeds both the judge template's `instructions` and the assertion, so editing169 it moves expected and actual together and the run stays green. Edit the input alone:170171 ```172 # in the spec, temporarily — inside the template `instructions` string only:173 # "explanation": "${verdict}-BROKEN saw {{output}}"174 bin/e2e test flows/evals/eval-task.spec.ts175 # → expect(received).toBe(expected)176 # Expected: "e2e-verdict-w0-m1x2y3 saw llm"177 # Received: "e2e-verdict-w0-m1x2y3-BROKEN saw llm"178 ```179180 Perturb the side under test, never a value shared by the assertion and the input — if the run181 stays green, the perturbation proved nothing. Quote the failure in your notes, restore the file,182 and run green again. A check that still passes perturbed is not a check.1831844. **Record the evidence**: the exact commands, the last green run's summary, the quoted failure per185 check, and the `E2E: new <ID>` (or `updated <ID>`) line for the PR body.186187### 5. Hand off188189Spec and `FLOWS.md` in the same commit. No `test.only` — CI runs `forbidOnly`. No retries: a flow190that passes on the second attempt is telling you something true. Never pass `--grep-invert` on the191command line; it replaces the config's own value and un-quarantines the whole suite. Quarantine only192with an owner, a ticket and an expiry within 45 days. The PR body carries the evidence from step 4.193194State plainly which of step 4's runs you did not do, and why. A flow handed over without its managed195run or its fallibility proof is a draft; naming that is what stops a reviewer reading it as coverage.196197## Quick reference198199Signatures, the pinned app facts (endpoints, page sizes, wire bodies, grid and filter selectors, the200in-container gateway address), the mock-LLM contract and the ClickHouse/Postgres binding traps are in201[references/harness-cheatsheet.md](references/harness-cheatsheet.md) — read it while writing the202spec.203204| Primitive | Import from | Use it for |205| ---------------- | ----------------------- | ---------------------------------------------- |206| `test`, `expect` | `../../lib/fixtures` | every flow that starts signed in |207| `test as base` | `@playwright/test` | only a flow that drives login or signup itself |208| `actor` fixture | `../../lib/fixtures` | worker-scoped org, keys, authed `ApiClient` |209| `probe` fixture | `../../lib/fixtures` | `ch` / `pg` / `apiList` reads |210| `sendTrace` | `../../lib/otlp` | seeding observe data through the collector |211| `POLL` | `../../lib/state-probe` | the shared wait budgets below |212| `E2E` | `../../lib/env` | endpoints — never hardcode a port |213| `flowAnnotation` | `../../lib/flow-meta` | the catalog contract |214215Pick a budget by what you are waiting on, never by which flow you are writing:216217| Budget | Waits on |218| ------------------------------------ | ---------------------------------------------------------------------------------------------- |219| `POLL.SPAN_VISIBLE` (15 s) | anything fi-collector writes — `spans`, curated `traces` |220| `POLL.EVAL_RESULT` (90 s) | a Temporal eval task reaching a terminal status |221| `POLL.CDC_VISIBLE` (180 s) | any CDC-fed table: `tracer_eval_logger`, `model_hub_score`, datasets, prompts, simulate, usage |222| `POLL.ASYNC_JOB` (60 s) | other Temporal work |223| `UI_READY` (60 s, per-spec constant) | every browser `expect`, `toHaveURL`, `waitForResponse` |224225## Red flags — STOP226227| Thought | Reality |228| ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |229| "I'll add the annotation later." | `@flow` without an annotation breaks `yarn catalog` for the whole suite; the catalog is CI's first gate, before the stack boots. |230| "`toContainText` is fine here." | It passes with the feature broken. The README's rule is exact sets: `toHaveText([alpha])` proves the other rows are gone. |231| "I'll raise the project timeout." | That slows every flow for one flow's budget. Raise this test's own ceiling with `test.setTimeout`, sized as the sum of the chained budgets. |232| "Skip the managed run, CI will catch it." | CI builds the frontend from source, but a backend, collector or gateway your PR does not touch runs as the released `:latest`. A flow depending on an unreleased change to one of those is not exercised there at all. |233| "This spec has never been executed; the first run will confirm the locators." | Then the first run is the author's, not the reviewer's. An unexecuted spec is a draft, and the plan you agreed to said "proven". |234| "It typechecks, it collects, the catalog renders — it's verified." | All three pass on a flow whose every assertion is wrong. They check the shape, not the claim. Only step 4.3 checks the claim. |235| "Assert on the first row / on the count." | `actor` is shared by every spec in the worker and nothing is ever deleted. Anchor on a value you minted; measure a delta where prior rows exist. |236| "The mock can just return X." | It cannot be programmed. It echoes the last user message with fixed usage. Shape the prompt so the echo parses — see the cheatsheet's recipe. |237| "Use `127.0.0.1`." | Login skips reCAPTCHA only when the request `Host` contains `localhost`. Address the stack as `localhost`. |238| "I'll add the `data-testid` in this PR." | Grep first — the hook you need usually exists. If it genuinely does not, the README still wants the attribute as its own small product PR, and your local managed run uses the released frontend until you `bin/e2e build frontend`. CI does build the frontend from source, so a same-PR attribute is exercised there. |239| "There's no `aria-pressed`, so I'll assert the computed style — I check the unselected one too, so it can still fail." | That answers a different objection. A locator bound to a CSS value breaks on any restyle and cannot be verified without the stack. It is the `data-testid` case; treat it as one. |240| "One retry won't hurt." | `retries: 0` is deliberate. A flow that passes on attempt two is reporting a real product behaviour; a retry throws that signal away. Quarantine with owner, ticket and expiry instead. |241| "I'll read the spans table without `FINAL`." | `spans`, `traces` and every CDC-fed table are ReplacingMergeTree. An unmerged read hands you the stale version of a row that moved through a lifecycle. |242| "This wait is a UI refresh, not an eventually-consistent store, so the default budget is right." | The 10 s expect default is not a first-paint budget. A UI step that takes 10 s alone takes a minute beside other workers. Use the spec's `UI_READY`. |243| "`FLOWS.md` is generated text; I'll hand-produce what the generator would emit." | The gate is a byte compare, and the file says "do not edit" in its own header. Run `yarn catalog`. |244| "`login.spec.ts` bypasses the fixtures, so I can assemble the actors I need by hand." | That file is a documented single-actor exception because it drives the login UI. Multiple identities is a harness gap: propose the `lib/` extension, do not build it inside a flow. |245| "The product's own path doesn't reach that state, so I'll drive the API instead." | You have just changed what the flow proves. That is an interview question, not an implementation detail. |246| "The commit is mostly refactor; I know which claim is the user-visible one." | Two agents reading the same commit picked two different goals. If more than one goal is plausible, ask. |247| "The diff tells me what changed; I don't need the ticket." | The diff says what changed. The ticket says what a user can now do — which is what the flow asserts, and what its `userGoal` has to state. |248249## Common mistakes250251Read [references/footguns.md](references/footguns.md) when a run fails for a reason you did not252predict — it is the full symptom → cause → fix list. The eight that bite first:253254- Title is not exactly `<ID>: <sentence>` — the catalog renders the id twice and quarantine by id255 stops matching.256- `area` in the annotation does not equal the `flows/<area>/` directory — nothing validates it; it257 silently opens a new section in `FLOWS.md`.258- Forgetting `yarn catalog` after touching a title, tag or annotation — CI fails before boot.259- `spans.trace_id` bound as anything but `{t:String}`, or curated `traces` looked up by a260 `trace_id` column that does not exist (it is keyed by `id`, bound `{t:UUID}`).261- ClickHouse `count()` compared as a number — it comes back a string; wrap it in `Number(...)`.262- `probe.apiList` used on an endpoint that does not return `{ result: { table } }` — it hard-codes263 that envelope.264- In-container callers given a host URL — the eval judge runs in the worker container and must265 address the gateway over the compose network.266- `yarn typecheck` skipped — CI does not run it, and Playwright strips types without checking them.267268## Out of scope269270- Retrofitting `data-testid` across product code in this PR. One attribute, its own small PR, only271 where the semantics are genuinely ambiguous.272- Extending `e2e/lib/` from inside a spec. A new fixture, probe or seeder is its own piece of work,273 planned with the user.274- Flows for services this stack profile does not start — `serving`, `code-executor`, the simulation275 runner — or assertions on surfaces whose CDC mirror is known-drifted.276- Committing. Prepare the spec, the regenerated `FLOWS.md` and the evidence; the user decides when277 it lands.
future-agi