User-perspective E2E testing
Project-local skill for ScienceDiscovery. E2E means simulating actual
product use from a user's goal to an observable outcome, not selecting a
particular test framework. Browser journeys use Playwright; public API, CLI,
and local-stack journeys also qualify when they cross the real product entry
point. Package tests, type checks, in-process calls to internal functions, or
assertions that only prove a request/click occurred are not E2E.
Read the shared target/isolation rules for every journey. For browser work,
follow the pinned Playwright, metadata, fixtures, and report rules below. For
non-browser work, follow API / CLI / local-stack journeys.
pnpm ci:e2e and the CI e2e layer still run the mocked browser subset;
they neither discover nor certify every kind of user-perspective E2E.
The primary unit of coverage is a user journey: one recognizable user goal
from entering the product through its observable outcome. Do not split the
main suite into internal-module specs such as shell.spec.ts, python.spec.ts,
or environment.spec.ts. Shell, Python, managed environments, subagents,
permissions, and artifacts should appear naturally as steps inside journeys.
A journey is written as numbered user steps, and every run writes a
step-by-step report a human can read without opening the spec. That contract is
mandatory for new work. Browser automation uses Journey steps and automatic
reports; API/CLI drivers record
equivalent steps and outcomes without requiring page screenshots.
When to add coverage
When an implementation changes user-observable behavior, its implementer must
add or improve the relevant journey with the feature, unless existing coverage
already verifies the changed contract (identify and rerun it). Do not defer
journey coverage to PR preparation or another role. This includes Run outcomes,
tool execution, queryable versioned state, artifacts, permissions, failure
feedback, cancellation, and recovery, even when no page changes. An independent
tester, when assigned, designs scenarios from the user requirements rather
than copying implementation conclusions.
Choose the interface the user actually uses. API coverage complements but does
not replace browser checks for changed UI behavior. Cover success and the
relevant failure/repeat/cancel/recovery paths in proportion to risk.
Write E2E: not applicable only when there is no user-observable product
path affected (for example documentation/comments-only changes), and explain
why. Backend-only / no new UI is not an exemption. A missing environment
or journey driver is BLOCKED or a coverage gap, not “not applicable.”
Target commit and worktree
Keep the repository's main worktree read-only. Obtain the target ref from the
test or review request, fetch that ref only when it is not available locally,
then resolve and record its immutable commit SHA. Do not automatically merge,
rebase, or replace the requested target with another branch. Only when the
requested target is origin/master, fetch it immediately before resolving and
testing that fetched SHA.
Use the task's assigned worktree by default. Before a formal run, commit the
candidate and record its immutable SHA; do not alter tracked files while that
SHA is under test. Create a detached, run-specific worktree only when the task
worktree is being rebased, fault injection must mutate it, or parallel runs
cannot otherwise isolate their ports/data:
git worktree add --detach .worktrees/e2e-<change>-<run-id> <target-sha>
Keep the target commit unchanged. Assemble dependencies, services, data,
ports, and evidence only in the assigned or explicitly justified E2E worktree,
using ignored or run-specific paths. Report the requested target ref, resolved
target SHA, worktree, and whether the run used any separate test-only derived
commit.
If validation reveals that code, a spec, or this skill must change, stop
treating the current run as validation of the target. Create and commit a new
candidate under the repository's development rules, then rerun E2E against
that new immutable commit.
API / CLI / local-stack journeys
- Start the isolated, committed product with
./scripts/start-stack.sh --mode local,
or a documented equivalent product entry point when testing that
entry itself. Use --no-build only for artifacts built from the target SHA.
Follow Isolated stack per run for ports, data,
service URLs and authentication. Verify the services required by the journey;
check the Web entry only when the journey uses it.
- Drive the running product from a separate client through its supported HTTP
API or documented CLI. A representative goal is: create a Project/Session,
submit a request, handle any permission prompt, wait for the Run terminal
state, retrieve the output/artifact, and confirm the Session can continue.
Assert the documented result and feedback, not a hard-coded incidental tool
order. Neither importing
createNativeAgent/createAgentRun in-process nor
starting only a test-constructed server substitutes for the product startup
path. /api/health alone is a preflight, not this complete journey.
- Store reusable non-browser drivers in
test/api/, named after the user goal,
with their exact root-level invocation and required environment documented.
The existing run_m1_smoke.sh and run_real_smoke.sh directly instantiate
the adapter: they remain smoke/integration checks, not E2E by location or
name. There is currently no generic non-browser E2E command; inspect the
committed driver or add the missing journey. Do not invent a pnpm entry
point or claim ci:e2e ran it. Integrate future drivers into the existing
test catalog when requested, without silently changing CI layer behavior.
- Declare the same purpose, steps, environment, external capabilities,
credentials, mocked/real choice and cost/side effects as E2E-META. Playwright
tags, fixtures and
check-e2e-meta.mjs apply to browser specs; a non-browser
driver must document and enforce its own gates and reporting. Default to a
journey-owned local model stub registered through the product API. Real
models/services require explicit opt-in and documented costs. Browser
network guards do not protect a CLI or backend: keep their configured
endpoints local and audit all external calls as well.
- Wait for observable status with bounded polling/stream consumption, not
fixed sleeps. Record numbered user steps, expected/actual results, SHA,
startup and driver commands, health, redacted request/response summaries,
CLI exit status/output, and necessary service logs in
report.md for PASS,
FAIL and BLOCKED. Assert product-promised persistent records where relevant;
internal database reads alone do not replace the user's access path. A
driver must fail on a failed outcome and report missing prerequisites as
BLOCKED, never as a successful zero-case run. No fake screenshots required.
- Clean up only journey-owned records and processes in
finally, after
preserving evidence. Keep data and report files in the run's isolated
directories; never publish credentials or commit generated results.
Browser: the pinned Playwright, and nothing else
Use only the @playwright/test declared by test/e2e.package.json, assembled
into the gitignored .e2e/ environment:
node test/sync-e2e.mjs --write
cd .e2e
npm install
./node_modules/.bin/playwright install chromium
Never use npx playwright, a globally installed Playwright, a throwaway npm
project, or system browser drivers. Every npm test/list command runs the
fail-closed sync-e2e.mjs --check first. If a committed manifest, lockfile, or
config differs from .e2e, synchronize and rerun npm install; never bypass
the check. Each worktree gets its own .e2e/; only the npm download cache and
the Playwright browser cache may be shared.
If Chromium exits at sandbox_host_linux.cc with Operation not permitted
inside an execution sandbox, classify it as test infrastructure and request an
approved run outside that sandbox. Do not work around it with extra unsafe
browser flags or a different browser binary.
Isolated stack per run
Run formal E2E from the task's assigned immutable worktree or the explicitly
justified detached worktree described above, never from the main worktree or
an unrelated development worktree.
One run owns one worktree + SHA, one set of API/Runner/Gateway processes and
ports, one data directory, and one artifact directory. Parallel runs must be
fully isolated or serialized. Start the stack from the worktree root:
./scripts/start-stack.sh --mode local # provisions and builds
./scripts/start-stack.sh --mode local --no-build # only if the build matches this SHA
Isolation variables (set for start-stack.sh and the API alike):
| Variable |
Default |
Meaning |
SCIENCE_AGENT_PORT |
4310 |
API/Web port |
SCIENCE_AGENT_RUNNER_PORT |
4311 |
where the runner listens |
SCIENCE_AGENT_RUNNER_URL |
http://127.0.0.1:<port> |
where the API dials — the API does not read SCIENCE_AGENT_RUNNER_PORT, so on a non-default port always set the full URL too |
SCIENCE_AGENT_DATA_DIR |
.sciencediscovery-data |
data root; also holds envs/ (service venvs) |
SCIENCE_AGENT_AUTH_TOKEN |
generated when omitted |
Optional server-side override; never assume a literal default |
E2E_API_TOKEN |
none |
Required browser/API token; use the value printed/generated by this isolated stack |
E2E_BASE_URL |
http://127.0.0.1:4310 |
must point at the same API the fixtures use |
UV_CACHE_DIR |
uv default (~/.cache/uv) |
Pin inside the run worktree (for example $SCIENCE_AGENT_DATA_DIR/uv-cache). The home cache sits outside the worktree and fails in environments that cannot write $HOME/.cache. |
After starting, verify API/Runner and any other service required by the
journey; verify the web entry for browser journeys. Export the stack's token
as E2E_API_TOKEN; browser global setup fails before the first scenario when
it is absent. Non-browser clients must use the same stack and authentication.
When a required service,
port, or credential is missing, record the run as BLOCKED with the missing
item — never skip silently.
Isolation traps that produce confusing failures:
- Never share
.sciencediscovery-data/envs between worktrees (no symlinks). The service
venvs are editable installs: their .pth files point back at the source
tree of whichever worktree provisioned them, so a shared venv silently runs
another worktree's code. Provision per worktree; uv sync is fast when
UV_CACHE_DIR points at a worktree-local cache.
- Pin
UV_CACHE_DIR inside the worktree. uv's default cache is
~/.cache/uv. Export UV_CACHE_DIR to a run-owned path (for example
$SCIENCE_AGENT_DATA_DIR/uv-cache) before start-stack.sh / uv sync. A
permission error against the home cache is test infrastructure, not a
product defect. npm and Playwright browser caches may still be shared as
above; do not send uv's cache to $HOME.
- Export every isolation variable explicitly. A worktree under
.worktrees/ lives inside the main repository, and the gateway's dotenv
loading searches parent directories, so the main repo's .env (its ports,
data dir, model config) leaks into the gateway even when the worktree has no
.env. Explicitly exported variables win over dotenv; alternatively copy a
trimmed .env into the worktree as the run's baseline.
Browser: discover, filter, run
For repository CI scheduling, use the cross-runner catalog from the repository
root before assembling a stack:
pnpm ci:tags
pnpm ci:list -- --tag layer:e2e --tag llm:stub --tag arch:amd64
pnpm ci:run -- --case e2e.mocked
Every catalog case declares arch:*, llm:*, npu:*, and sandbox:* plus
layer/container/network tags. Repeated --tag clauses mean AND; comma-separated
tags within one clause mean OR; --exclude removes matches. Keep e2e.mocked,
e2e.real, and e2e.legacy aligned with the Playwright projects below whenever
their requirements change. Never reclassify an unaudited dependency as safe:
use an unreviewed tag or keep the case unsupported until evidence exists.
The catalog chooses a CI-capability group; Playwright still discovers and
filters the individual specs in that group. Use the pinned .e2e commands
below for file/title-level discovery.
Run from .e2e/ after synchronizing and installing the committed environment:
npm run check:meta # validate metadata, tags, quarantine
npm run test:list # list only @mocked (safe default)
npm test # run only @mocked (same as below)
npm run test:mocked # run only @mocked
npm run test:real:list # discover only @real; makes no live call
npm run test:real # RUN live @real (explicit opt-in)
npm run test:mixed:list # discover @mocked + @real
npm run test:mixed # RUN both groups (explicit opt-in)
npm run test:mocked -- foo.spec.ts # one file, relative to test/
npm run test:mocked -- -g "title" # semantic title filter
The mocked group is credential-free, not stack-free: start the isolated stack
before the default suite. Only explicitly self-contained specs such as the
network guard may run without it.
Grouping is enforced by the Playwright projects in
test/playwright.config.ts:
mocked — only specs tagged @mocked. It is the sole default project and
uses a journey-owned local stub model to drive deterministic, user-visible
product contracts (for example permission feedback, tool process, artifacts,
and environment state). It must be hermetic, credential-free, stable, and
repeatable. Its custom test fixture installs the HTTP(S)/WebSocket guard
before hooks or pages. The project also blocks service workers so they
cannot bypass routing.
real — specs tagged @real. The project is defined only when E2E_REAL=1
is set, so no default command can trigger live LLM calls, network access, or
paid usage. Without the variable, --project=real fails with "Project(s)
'real' not found": that run is BLOCKED, not passed. Keep this group small:
it is a smoke check that a real user can express the goal naturally and
reach the outcome, not a duplicate deterministic matrix. Assert stable user
invariants, never exact model wording or a single incidental tool sequence.
legacy — untagged specs listed in LEGACY in
test/check-e2e-meta.mjs. They are quarantined because their external
behavior has not been audited. Use npm run test:legacy:list to inventory;
run npm run test:legacy only with explicit approval and the same caution as
real tests. During initial adoption, only pre-existing unaudited specs from
the base branch may enter LEGACY; never place a newly written journey there.
Migrate the list incrementally without making old specs undiscoverable.
test:real and test:mixed are execution opt-ins, not discovery commands.
Before using either, inspect E2E-META, confirm credentials/endpoints/costs and
record authorization. A skipped real test with a BLOCKED: reason remains
BLOCKED in the run report; zero failures does not turn blocked/skipped cases
into PASS.
E2E-META: every browser test documents itself
Each test() carries an E2E-META comment directly above it and a matching
{ tag: "@mocked" } or { tag: "@real" } option. node test/check-e2e-meta.mjs
enforces the fields and tag consistency (legacy files are warned until
migrated; new files must comply). Template:
/**
* E2E-META
* Purpose: <the user-visible behavior this verifies>
* Steps:
* 1. <main step>
* 2. <main step>
* Environment: <stack, base URL, services, ports, data/fixture preconditions>
* Type: mocked | real
* LLM: <none | local stub | provider/model/endpoint and nondeterminism>
* WebSearch: <none | engine/endpoint and expected queries>
* PaperSources: <none | PubMed/arXiv/etc. and expected queries/downloads>
* MCP: <none | server/tools and expected calls>
* OtherExternal: <none | any other network/process/service behavior>
* Credentials: <none | exact env vars or seeded configuration>
* CostSideEffects: <none | fees, rate limits, writes, messages, mutations>
*/
Mocked example:
/**
* E2E-META
* Purpose: A hung agent turn times out with a user-visible reason.
* Steps:
* 1. Register a silent local stub model over the API.
* 2. Start a run and wait for the idle timeout.
* 3. Assert the timeout reason in the session view.
* Environment: Running stack at E2E_BASE_URL; isolated data; no models.
* Type: mocked
* LLM: local silent HTTP stub only; no live model.
* WebSearch: none
* PaperSources: none
* MCP: none
* OtherExternal: none — non-local browser requests are aborted.
* Credentials: none
* CostSideEffects: no cost; temporary records are deleted in finally.
*/
test("超时原因可追溯", { tag: "@mocked" }, async ({ page }) => {
Real example:
/**
* E2E-META
* Purpose: A live model run produces an anchored plan card.
* Steps:
* 1. Register the real model via env config; create project/session.
* 2. Run a planning prompt; assert the card position.
* Environment: Isolated stack at E2E_BASE_URL; E2E_SCREENSHOTS for output.
* Type: real
* LLM: real chat completions via E2E_LLM_BASE_URL; output varies.
* WebSearch: none
* PaperSources: none
* MCP: none
* OtherExternal: none
* Credentials: E2E_LLM_BASE_URL / E2E_LLM_MODEL / E2E_LLM_TOKEN.
* CostSideEffects: billable tokens and provider rate limits; creates isolated records.
*/
test("plan card anchors", { tag: "@real" }, async ({ page }, testInfo) => {
requireRealEnv(testInfo, "E2E_LLM_BASE_URL", "E2E_LLM_MODEL", "E2E_LLM_TOKEN");
await requireRealStack(testInfo);
// Check any seeded model/connector state here and testInfo.skip(true, "BLOCKED: ...")
// only when that identifiable precondition is absent.
await page.goto("/");
// Product assertions after satisfied gates still FAIL normally.
});
Journey steps and automatic reports (mandatory)
Every test/journey-*.spec.ts must be written as user steps through the
journey fixture. This is a requirement, not a suggestion:
node test/check-e2e-meta.mjs fails a journey spec that does not request the
{ journey } fixture, does not call journey.scenario(...), does not call
journey.step(...), or takes its own page.screenshot().
The rules
- One
journey.step() = one user step. Use the steps from the journey's
design document. Merge adjacent micro-interactions (fill three fields, then
save) into the step a user would name; never split by internal module and
never make one step per click.
- The step description says what the user should see, in the reader's
language, not what the code asserts. Someone who has never read the spec
must be able to follow the report.
- Every step's evidence comes from the helper. It waits for the page to
settle, screenshots, and files that step's console/network noise. Do not
hand-roll
page.screenshot("01-...") as primary evidence.
journey.scenario({ goal, preconditions }) is required, at the top of
the test. It supplies the report's goal and preconditions; without it the
report falls back to the English E2E-META Purpose/Environment, which is
contract text rather than an explanation for a reader.
- A report is written for every outcome — passed, failed, and blocked —
because the fixture tears the reporter down. Steps that already ran stay in
the report when a later step fails, and the failing step keeps its own
screenshot plus an error summary.
- Do not weaken an assertion to make a report green. A journey that
documents a known product gap stays FAIL and says so in its preconditions.
What a run produces
.e2e/journey-reports/<spec>/<test>/ (gitignored) receives:
| File |
Content |
report.md |
Result, commit SHA under test (and whether the tree was dirty), spec file, group/tags, start/end/duration, scenario goal, preconditions, gate reason when blocked, the step table, and per-step detail |
report.html |
The same content, self-contained: inline CSS only, screenshots by relative path, opens from the filesystem with no network |
NN-<step>.png |
One screenshot per step, in step order; a step that failed or was blocked is suffixed accordingly |
Per-step "key logs" are browser console errors/warnings, page errors, failed
requests, and any response with status ≥ 400 — recorded as method, URL, and
status. Bodies are never captured. Every recorded string passes through
redaction: known credential variables, Bearer values, token=/api_key=
query parameters, the repository root, and the home directory are replaced. Keep
it that way; reports get published.
Both files are also attached to the Playwright HTML report, so a CI run carries
them without a separate copy step.
Set E2E_JOURNEY_REPORTS to redirect the output root. Never commit reports,
screenshots, or .e2e/.
Shortest complete example
import { expect } from "@playwright/test";
import { test } from "./helpers/e2e.ts";
import { createProjectAndSession, openProjectSession, scriptedModel,
sendUserMessage, waitForRunTerminal, expandToolStep, cleanupJourney } from "./helpers/journeys.ts";
/**
* E2E-META
* ... all eleven fields ...
*/
test("J9 用户可以拿到一次可核对的执行结果", { tag: "@mocked" }, async ({ journey, page }) => {
journey.scenario({
goal: "一位研究员要确认产品真的在本机执行了他要求的计算,而不只是在聊天。",
preconditions: ["隔离栈已启动", "模型由旅程自带的本地 stub 驱动,不访问外部服务"],
});
const marker = `J9-${Date.now()}`;
const stub = await scriptedModel([
{ arguments: { command: `echo ${marker}` }, tool: "run_shell" },
{ text: "The requested check completed." },
]);
const fixture = await createProjectAndSession(page, {
model: { apiToken: stub.apiToken, baseUrl: stub.baseUrl, model: stub.model, name: `J9 ${Date.now()}` },
projectName: `J9 ${Date.now()}`,
sessionTitle: `J9 ${Date.now()}`,
});
try {
await journey.step(
"进入会话并提出请求",
"任务发出后运行走到完成态,主按钮回到可再次运行的状态。",
async () => {
await openProjectSession(page, fixture);
const run = await sendUserMessage(page, fixture.session.id, "Check the workspace and report what you find.");
expect((await waitForRunTerminal(page, fixture.session.id, run.id)).status).toBe("completed");
},
);
await journey.step(
"展开工具步骤核对输出",
"工具步骤展开后能看到这次执行的真实输出,说明命令确实跑过。",
async () => {
await expect(await expandToolStep(page, { contains: marker })).toContainText(marker);
},
);
} finally {
await cleanupJourney(page, fixture);
await stub.stop();
}
});
A @real journey follows the same shape; put requireRealEnv(...) and
await requireRealStack(...) inside the first journey.step, so a run
without credentials still produces a report whose first step reads
⛔ 前置未满足 with the gate reason.
Migration pace for existing specs
Specs quarantined in LEGACY may adopt this incrementally; they are not
blocked on it. A new file has no grace period: any spec added as
journey-*.spec.ts complies on its first commit, and a non-journey spec that
grows into a full user flow should be renamed and converted rather than kept as
an untracked exception.
Browser mocked rules
- The model is always a spec-owned local stub (
node:http createServer on
127.0.0.1, registered through /api/models) or a seeded fake such as the
hang/slow models. Never read E2E_LLM_* in a mocked spec, and never rely on
a real model configured in the stack's .env.
- Import
test from test/helpers/e2e.ts, never directly from
@playwright/test. Its automatic fixture aborts non-local HTTP(S),
policy-closes non-local WebSockets, and forwards localhost/loopback traffic
before any beforeEach, navigation, or request. check-e2e-meta.mjs
enforces this import for every Type: mocked file, including fixme tests
when later enabled.
- Keep
test/e2e-network-guard.spec.ts passing as the request-level proof for
blocked HTTPS/WebSocket and allowed local HTTP/WebSocket behavior.
- Backend egress cannot be intercepted from the browser; it stays local
because the only model the spec registers is its own stub. When a mocked
spec must prove a call happened, assert on the stub (request counters,
captured bodies), not on timing.
- Mocked specs must pass repeatedly on a clean stack with no credentials.
Browser real rules
- Real specs are explicit opt-in (
E2E_REAL=1), tagged @real, and declare
their external behavior and cost in E2E-META.
- At the start of each real test body, before
beforeEach-equivalent
navigation, API setup, LLM, search, paper-source, or MCP actions, call
requireRealEnv(testInfo, "E2E_LLM_BASE_URL", "E2E_LLM_MODEL", "E2E_LLM_TOKEN")
and await requireRealStack(testInfo). Do not put the gate only at file
scope. check-e2e-meta.mjs enforces both calls per real test body and their
order relative to the first external action.
- If a real test genuinely needs no standard LLM credentials, call
allowRealEnvException(testInfo, "specific reviewed reason"); the stack
gate remains mandatory and the exception appears in test annotations.
- For seeded runtime state (for example, model registry or PubMed connector),
call
testInfo.skip(true, "BLOCKED: ...") with a specific reason before first use only
when that identifiable precondition is absent. Once gates pass, product
assertion failures must remain FAIL. Report all gate skips as BLOCKED, never
as passes.
- Expect nondeterminism: assert user-visible invariants, not exact model
output; use generous observable-condition waits, not fixed sleeps.
- Record what the run actually consumed (model, endpoints, connectors) and
keep traces/screenshots so results can be reviewed without re-spending.
Writing browser specs
Organize files and describe blocks by user goal, not by implementation
module. A journey may use shell, Python, an environment, a subagent, and
artifact versioning in one coherent flow. Small negative/cancel/recovery
cases may sit beside the journey they qualify. Name main-flow files after
the goal (journey-first-run.spec.ts, journey-deliver-result.spec.ts),
never after an internal tool (shell.spec.ts, python.spec.ts).
Use test/helpers/journeys.ts for common user actions: model registration
and selection, Project/Session setup, natural-language submission, Run
terminal-state waiting, permission handling, timeline/tool-process reading,
environment revision lookup, and opening the environment or artifact
surfaces. Use scriptedModel(mainSteps, subagentSteps?) for deterministic
tool/text sequences; it routes general-purpose subagent requests by their
preset system marker. Use expandToolStep(page, { contains }) with a
journey marker instead of making run_shell or run_python the only
locator. artifactTree(page) deliberately exposes the declared catalog,
@ candidates, and the opt-in physical workspace tree as separate views.
The helpers return records, locators, and visible text; the spec still owns
goal-specific assertions.
Use the journey fixture from test/helpers/e2e.ts (implemented in
test/helpers/journey-report.ts) to structure the test as user steps and
produce its report. See Journey steps and automatic
reports for the full
contract and a copyable skeleton.
Prefer semantic locators (getByRole, getByLabel, getByText); when only
a CSS hook works, that is also an accessibility signal worth noting.
Wait on observable conditions (toast visible, button state, request
finished); never sleep-and-hope. Assert user-visible outcomes, not merely
that a request or click happened.
Create test data with unique names (Date.now() suffix) and clean up in
finally where practical; leftover data must stay in the run's own data
directory.
Long-term regression specs live in test/; throwaway diagnostic specs stay
in the E2E worktree and are never committed.
Browser screenshots, evidence, artifacts
- Journey specs take no screenshots of their own.
journey.step() names
them in step order, waits for the page to settle, and files them with the
step they belong to. Add evidence by adding or resplitting a step, never by
calling page.screenshot() — check-e2e-meta.mjs rejects that.
- Non-journey specs that still need an ad-hoc shot write to
screenshots/
relative to the Playwright cwd (i.e. .e2e/screenshots/), or to
E2E_SCREENSHOTS when the spec supports it, named in step order.
- Shoot key success and failure states only after the target text/state is
stable; avoid skeletons, animation remnants, and clipped controls. Journey
screenshots already pass Playwright
animations: "disabled" (and
caret: "hide") from journey.step(). Ad-hoc page.screenshot() calls
must set the same animations: "disabled" option; omitting it leaves CSS
transition ghosts in otherwise stable shots. Eyeball every screenshot after
the run; retake any that does not match its caption — a screenshot that
contradicts its step description is a report defect.
- Failures keep the automatic failure screenshot and trace
(
trace: retain-on-failure); reports land in playwright-report/ and
test-results/ (including results.json) next to the config in use. Journey
reports are additionally attached to each test in the Playwright HTML report.
- A run report references evidence by relative path inside the run's artifact
directory and states: SHA under test, worktree, ports, config baseline,
start and test commands, health status, per-case expected/actual, and
failure attribution. Report discovered, executed, passed, failed, and
skipped/fixme counts separately; discovered or skipped tests are not passes.
- Never commit
.e2e/ (which contains journey-reports/),
test/node_modules, test/playwright-report/, test/test-results/,
screenshots, traces, or logs — all gitignored. Committed files are the specs,
test/e2e.package.json + test/e2e.package-lock.json,
test/playwright.config.ts, test/helpers/ (including
helpers/journey-report.ts), test/sync-e2e.mjs, and
test/check-e2e-meta.mjs.
- To hand a report to a reviewer, copy
report.md, report.html, and that
directory's .png files out as a unit — the HTML references the images by
relative name, so the folder must stay together.
Failure attribution
Prefer negative and fault injections that do not modify tracked files. Keep
the target worktree read-only. If tracked changes are unavoidable, use a
separate worktree derived from the target and create a test-only commit or an
exact index/patch snapshot before injection. Restore from that snapshot and
report the derived commit when present; never pollute the target candidate.
| Type |
Judgment |
| Product defect |
UI/API/Gateway/Runner/orchestration breaks the user goal or gives no reasonable feedback |
| External environment |
A third-party dependency failed and the product prompted or degraded correctly |
| Test defect |
Selector, wait, timeout, fixture, or assertion is wrong; the product behaves |
| Test infrastructure |
Worktree/port/data clash, wrong service, build ≠ SHA, dependency assembly or discovery error |
A timeout is a tripwire, not a conclusion: confirm whether the run was issued,
the service responded, and the UI/API/CLI gave feedback. Every verdict needs at least
one piece of evidence (network log, console, service log, product message,
screenshot, trace); timeouts need two points along that chain. An external
failure the product swallows silently is still a product defect. Fix test or
infra problems and rerun; record BLOCKED when the cause cannot be separated.
Cleanup
Archive evidence first, then stop the stack, free the ports, and remove run
data and .e2e/ from the assigned worktree. Do not remove the task worktree;
its lifecycle belongs to the task owner. Remove a separate detached E2E
worktree only when this run created it for one of the justified isolation
reasons above. The main worktree must remain untouched.
1---2name: e2e-testing3description: Design, extend, and run ScienceDiscovery E2E user journeys through the browser, public HTTP API, CLI, or local product stack. Use when changing user-observable behavior, selecting mocked/real journeys, isolating a test stack, reporting user outcomes, or attributing an E2E failure. Browser journeys use the pinned Playwright environment in test/.4---56# User-perspective E2E testing78Project-local skill for **ScienceDiscovery**. E2E means simulating actual9product use from a user's goal to an observable outcome, not selecting a10particular test framework. Browser journeys use Playwright; public API, CLI,11and local-stack journeys also qualify when they cross the real product entry12point. Package tests, type checks, in-process calls to internal functions, or13assertions that only prove a request/click occurred are not E2E.1415Read the shared target/isolation rules for every journey. For browser work,16follow the pinned Playwright, metadata, fixtures, and report rules below. For17non-browser work, follow [API / CLI / local-stack journeys](#api--cli--local-stack-journeys).18`pnpm ci:e2e` and the CI `e2e` layer still run the mocked **browser subset**;19they neither discover nor certify every kind of user-perspective E2E.2021The primary unit of coverage is a **user journey**: one recognizable user goal22from entering the product through its observable outcome. Do not split the23main suite into internal-module specs such as `shell.spec.ts`, `python.spec.ts`,24or `environment.spec.ts`. Shell, Python, managed environments, subagents,25permissions, and artifacts should appear naturally as steps inside journeys.2627A journey is written as **numbered user steps**, and every run writes a28step-by-step report a human can read without opening the spec. That contract is29mandatory for new work. Browser automation uses [Journey steps and automatic30reports](#journey-steps-and-automatic-reports-mandatory); API/CLI drivers record31equivalent steps and outcomes without requiring page screenshots.3233## When to add coverage3435When an implementation changes user-observable behavior, its implementer must36add or improve the relevant journey with the feature, unless existing coverage37already verifies the changed contract (identify and rerun it). Do not defer38journey coverage to PR preparation or another role. This includes Run outcomes,39tool execution, queryable versioned state, artifacts, permissions, failure40feedback, cancellation, and recovery, even when no page changes. An independent41tester, when assigned, designs scenarios from the user requirements rather42than copying implementation conclusions.4344Choose the interface the user actually uses. API coverage complements but does45not replace browser checks for changed UI behavior. Cover success and the46relevant failure/repeat/cancel/recovery paths in proportion to risk.47Write **E2E: not applicable** only when there is no user-observable product48path affected (for example documentation/comments-only changes), and explain49why. **Backend-only / no new UI is not an exemption.** A missing environment50or journey driver is BLOCKED or a coverage gap, not “not applicable.”5152## Target commit and worktree5354Keep the repository's main worktree read-only. Obtain the target ref from the55test or review request, fetch that ref only when it is not available locally,56then resolve and record its immutable commit SHA. Do not automatically merge,57rebase, or replace the requested target with another branch. Only when the58requested target is `origin/master`, fetch it immediately before resolving and59testing that fetched SHA.6061Use the task's assigned worktree by default. Before a formal run, commit the62candidate and record its immutable SHA; do not alter tracked files while that63SHA is under test. Create a detached, run-specific worktree only when the task64worktree is being rebased, fault injection must mutate it, or parallel runs65cannot otherwise isolate their ports/data:6667```bash68git worktree add --detach .worktrees/e2e-<change>-<run-id> <target-sha>69```7071Keep the target commit unchanged. Assemble dependencies, services, data,72ports, and evidence only in the assigned or explicitly justified E2E worktree,73using ignored or run-specific paths. Report the requested target ref, resolved74target SHA, worktree, and whether the run used any separate test-only derived75commit.7677If validation reveals that code, a spec, or this skill must change, stop78treating the current run as validation of the target. Create and commit a new79candidate under the repository's development rules, then rerun E2E against80that new immutable commit.8182## API / CLI / local-stack journeys83841. Start the isolated, committed product with `./scripts/start-stack.sh --mode local`,85 or a documented equivalent product entry point when testing that86 entry itself. Use `--no-build` only for artifacts built from the target SHA.87 Follow [Isolated stack per run](#isolated-stack-per-run) for ports, data,88 service URLs and authentication. Verify the services required by the journey;89 check the Web entry only when the journey uses it.902. Drive the running product from a separate client through its supported HTTP91 API or documented CLI. A representative goal is: create a Project/Session,92 submit a request, handle any permission prompt, wait for the Run terminal93 state, retrieve the output/artifact, and confirm the Session can continue.94 Assert the documented result and feedback, not a hard-coded incidental tool95 order. Neither importing `createNativeAgent`/`createAgentRun` in-process nor96 starting only a test-constructed server substitutes for the product startup97 path. `/api/health` alone is a preflight, not this complete journey.983. Store reusable non-browser drivers in `test/api/`, named after the user goal,99 with their exact root-level invocation and required environment documented.100 The existing `run_m1_smoke.sh` and `run_real_smoke.sh` directly instantiate101 the adapter: they remain smoke/integration checks, not E2E by location or102 name. There is currently no generic non-browser E2E command; inspect the103 committed driver or add the missing journey. Do not invent a `pnpm` entry104 point or claim `ci:e2e` ran it. Integrate future drivers into the existing105 test catalog when requested, without silently changing CI layer behavior.1064. Declare the same purpose, steps, environment, external capabilities,107 credentials, mocked/real choice and cost/side effects as E2E-META. Playwright108 tags, fixtures and `check-e2e-meta.mjs` apply to browser specs; a non-browser109 driver must document and enforce its own gates and reporting. Default to a110 journey-owned local model stub registered through the product API. Real111 models/services require explicit opt-in and documented costs. Browser112 network guards do not protect a CLI or backend: keep their configured113 endpoints local and audit all external calls as well.1145. Wait for observable status with bounded polling/stream consumption, not115 fixed sleeps. Record numbered user steps, expected/actual results, SHA,116 startup and driver commands, health, redacted request/response summaries,117 CLI exit status/output, and necessary service logs in `report.md` for PASS,118 FAIL and BLOCKED. Assert product-promised persistent records where relevant;119 internal database reads alone do not replace the user's access path. A120 driver must fail on a failed outcome and report missing prerequisites as121 BLOCKED, never as a successful zero-case run. No fake screenshots required.1226. Clean up only journey-owned records and processes in `finally`, after123 preserving evidence. Keep data and report files in the run's isolated124 directories; never publish credentials or commit generated results.125126## Browser: the pinned Playwright, and nothing else127128Use only the `@playwright/test` declared by `test/e2e.package.json`, assembled129into the gitignored `.e2e/` environment:130131```bash132node test/sync-e2e.mjs --write133cd .e2e134npm install135./node_modules/.bin/playwright install chromium136```137138Never use `npx playwright`, a globally installed Playwright, a throwaway npm139project, or system browser drivers. Every npm test/list command runs the140fail-closed `sync-e2e.mjs --check` first. If a committed manifest, lockfile, or141config differs from `.e2e`, synchronize and rerun `npm install`; never bypass142the check. Each worktree gets its own `.e2e/`; only the npm download cache and143the Playwright browser cache may be shared.144145If Chromium exits at `sandbox_host_linux.cc` with `Operation not permitted`146inside an execution sandbox, classify it as test infrastructure and request an147approved run outside that sandbox. Do not work around it with extra unsafe148browser flags or a different browser binary.149150## Isolated stack per run151152Run formal E2E from the task's assigned immutable worktree or the explicitly153justified detached worktree described above, never from the main worktree or154an unrelated development worktree.155156One run owns one worktree + SHA, one set of API/Runner/Gateway processes and157ports, one data directory, and one artifact directory. Parallel runs must be158fully isolated or serialized. Start the stack from the worktree root:159160```bash161./scripts/start-stack.sh --mode local # provisions and builds162./scripts/start-stack.sh --mode local --no-build # only if the build matches this SHA163```164165Isolation variables (set for `start-stack.sh` and the API alike):166167| Variable | Default | Meaning |168|---|---|---|169| `SCIENCE_AGENT_PORT` | `4310` | API/Web port |170| `SCIENCE_AGENT_RUNNER_PORT` | `4311` | where the runner listens |171| `SCIENCE_AGENT_RUNNER_URL` | `http://127.0.0.1:<port>` | **where the API dials** — the API does not read `SCIENCE_AGENT_RUNNER_PORT`, so on a non-default port always set the full URL too |172| `SCIENCE_AGENT_DATA_DIR` | `.sciencediscovery-data` | data root; also holds `envs/` (service venvs) |173| `SCIENCE_AGENT_AUTH_TOKEN` | generated when omitted | Optional server-side override; never assume a literal default |174| `E2E_API_TOKEN` | none | Required browser/API token; use the value printed/generated by this isolated stack |175| `E2E_BASE_URL` | `http://127.0.0.1:4310` | must point at the same API the fixtures use |176| `UV_CACHE_DIR` | uv default (`~/.cache/uv`) | Pin inside the run worktree (for example `$SCIENCE_AGENT_DATA_DIR/uv-cache`). The home cache sits outside the worktree and fails in environments that cannot write `$HOME/.cache`. |177178After starting, verify API/Runner and any other service required by the179journey; verify the web entry for browser journeys. Export the stack's token180as `E2E_API_TOKEN`; browser global setup fails before the first scenario when181it is absent. Non-browser clients must use the same stack and authentication.182When a required service,183port, or credential is missing, record the run as **BLOCKED** with the missing184item — never skip silently.185186Isolation traps that produce confusing failures:187188- **Never share `.sciencediscovery-data/envs` between worktrees** (no symlinks). The service189 venvs are *editable* installs: their `.pth` files point back at the source190 tree of whichever worktree provisioned them, so a shared venv silently runs191 another worktree's code. Provision per worktree; `uv sync` is fast when192 `UV_CACHE_DIR` points at a worktree-local cache.193- **Pin `UV_CACHE_DIR` inside the worktree.** uv's default cache is194 `~/.cache/uv`. Export `UV_CACHE_DIR` to a run-owned path (for example195 `$SCIENCE_AGENT_DATA_DIR/uv-cache`) before `start-stack.sh` / `uv sync`. A196 permission error against the home cache is test infrastructure, not a197 product defect. npm and Playwright browser caches may still be shared as198 above; do not send uv's cache to `$HOME`.199- **Export every isolation variable explicitly.** A worktree under200 `.worktrees/` lives inside the main repository, and the gateway's dotenv201 loading searches parent directories, so the main repo's `.env` (its ports,202 data dir, model config) leaks into the gateway even when the worktree has no203 `.env`. Explicitly exported variables win over dotenv; alternatively copy a204 trimmed `.env` into the worktree as the run's baseline.205206## Browser: discover, filter, run207208For repository CI scheduling, use the cross-runner catalog from the repository209root before assembling a stack:210211```bash212pnpm ci:tags213pnpm ci:list -- --tag layer:e2e --tag llm:stub --tag arch:amd64214pnpm ci:run -- --case e2e.mocked215```216217Every catalog case declares `arch:*`, `llm:*`, `npu:*`, and `sandbox:*` plus218layer/container/network tags. Repeated `--tag` clauses mean AND; comma-separated219tags within one clause mean OR; `--exclude` removes matches. Keep `e2e.mocked`,220`e2e.real`, and `e2e.legacy` aligned with the Playwright projects below whenever221their requirements change. Never reclassify an unaudited dependency as safe:222use an `unreviewed` tag or keep the case unsupported until evidence exists.223224The catalog chooses a CI-capability group; Playwright still discovers and225filters the individual specs in that group. Use the pinned `.e2e` commands226below for file/title-level discovery.227228Run from `.e2e/` after synchronizing and installing the committed environment:229230```bash231npm run check:meta # validate metadata, tags, quarantine232npm run test:list # list only @mocked (safe default)233npm test # run only @mocked (same as below)234npm run test:mocked # run only @mocked235npm run test:real:list # discover only @real; makes no live call236npm run test:real # RUN live @real (explicit opt-in)237npm run test:mixed:list # discover @mocked + @real238npm run test:mixed # RUN both groups (explicit opt-in)239npm run test:mocked -- foo.spec.ts # one file, relative to test/240npm run test:mocked -- -g "title" # semantic title filter241```242243The mocked group is credential-free, not stack-free: start the isolated stack244before the default suite. Only explicitly self-contained specs such as the245network guard may run without it.246247Grouping is enforced by the Playwright projects in248`test/playwright.config.ts`:249250- `mocked` — only specs tagged `@mocked`. It is the sole default project and251 uses a journey-owned local stub model to drive deterministic, user-visible252 product contracts (for example permission feedback, tool process, artifacts,253 and environment state). It must be hermetic, credential-free, stable, and254 repeatable. Its custom `test` fixture installs the HTTP(S)/WebSocket guard255 before hooks or pages. The project also blocks service workers so they256 cannot bypass routing.257- `real` — specs tagged `@real`. The project is defined only when `E2E_REAL=1`258 is set, so no default command can trigger live LLM calls, network access, or259 paid usage. Without the variable, `--project=real` fails with "Project(s)260 'real' not found": that run is BLOCKED, not passed. Keep this group small:261 it is a smoke check that a real user can express the goal naturally and262 reach the outcome, not a duplicate deterministic matrix. Assert stable user263 invariants, never exact model wording or a single incidental tool sequence.264- `legacy` — untagged specs listed in `LEGACY` in265 `test/check-e2e-meta.mjs`. They are quarantined because their external266 behavior has not been audited. Use `npm run test:legacy:list` to inventory;267 run `npm run test:legacy` only with explicit approval and the same caution as268 real tests. During initial adoption, only pre-existing unaudited specs from269 the base branch may enter `LEGACY`; never place a newly written journey there.270 Migrate the list incrementally without making old specs undiscoverable.271272`test:real` and `test:mixed` are execution opt-ins, not discovery commands.273Before using either, inspect E2E-META, confirm credentials/endpoints/costs and274record authorization. A skipped real test with a `BLOCKED:` reason remains275BLOCKED in the run report; zero failures does not turn blocked/skipped cases276into PASS.277278## E2E-META: every browser test documents itself279280Each `test()` carries an `E2E-META` comment directly above it and a matching281`{ tag: "@mocked" }` or `{ tag: "@real" }` option. `node test/check-e2e-meta.mjs`282enforces the fields and tag consistency (legacy files are warned until283migrated; new files must comply). Template:284285```ts286/**287 * E2E-META288 * Purpose: <the user-visible behavior this verifies>289 * Steps:290 * 1. <main step>291 * 2. <main step>292 * Environment: <stack, base URL, services, ports, data/fixture preconditions>293 * Type: mocked | real294 * LLM: <none | local stub | provider/model/endpoint and nondeterminism>295 * WebSearch: <none | engine/endpoint and expected queries>296 * PaperSources: <none | PubMed/arXiv/etc. and expected queries/downloads>297 * MCP: <none | server/tools and expected calls>298 * OtherExternal: <none | any other network/process/service behavior>299 * Credentials: <none | exact env vars or seeded configuration>300 * CostSideEffects: <none | fees, rate limits, writes, messages, mutations>301 */302```303304Mocked example:305306```ts307/**308 * E2E-META309 * Purpose: A hung agent turn times out with a user-visible reason.310 * Steps:311 * 1. Register a silent local stub model over the API.312 * 2. Start a run and wait for the idle timeout.313 * 3. Assert the timeout reason in the session view.314 * Environment: Running stack at E2E_BASE_URL; isolated data; no models.315 * Type: mocked316 * LLM: local silent HTTP stub only; no live model.317 * WebSearch: none318 * PaperSources: none319 * MCP: none320 * OtherExternal: none — non-local browser requests are aborted.321 * Credentials: none322 * CostSideEffects: no cost; temporary records are deleted in finally.323 */324test("超时原因可追溯", { tag: "@mocked" }, async ({ page }) => {325```326327Real example:328329```ts330/**331 * E2E-META332 * Purpose: A live model run produces an anchored plan card.333 * Steps:334 * 1. Register the real model via env config; create project/session.335 * 2. Run a planning prompt; assert the card position.336 * Environment: Isolated stack at E2E_BASE_URL; E2E_SCREENSHOTS for output.337 * Type: real338 * LLM: real chat completions via E2E_LLM_BASE_URL; output varies.339 * WebSearch: none340 * PaperSources: none341 * MCP: none342 * OtherExternal: none343 * Credentials: E2E_LLM_BASE_URL / E2E_LLM_MODEL / E2E_LLM_TOKEN.344 * CostSideEffects: billable tokens and provider rate limits; creates isolated records.345 */346test("plan card anchors", { tag: "@real" }, async ({ page }, testInfo) => {347 requireRealEnv(testInfo, "E2E_LLM_BASE_URL", "E2E_LLM_MODEL", "E2E_LLM_TOKEN");348 await requireRealStack(testInfo);349 // Check any seeded model/connector state here and testInfo.skip(true, "BLOCKED: ...")350 // only when that identifiable precondition is absent.351 await page.goto("/");352 // Product assertions after satisfied gates still FAIL normally.353});354```355356## Journey steps and automatic reports (mandatory)357358Every `test/journey-*.spec.ts` **must** be written as user steps through the359`journey` fixture. This is a requirement, not a suggestion:360`node test/check-e2e-meta.mjs` fails a journey spec that does not request the361`{ journey }` fixture, does not call `journey.scenario(...)`, does not call362`journey.step(...)`, or takes its own `page.screenshot()`.363364### The rules3653661. **One `journey.step()` = one user step.** Use the steps from the journey's367 design document. Merge adjacent micro-interactions (fill three fields, then368 save) into the step a user would name; never split by internal module and369 never make one step per click.3702. **The step description says what the *user* should see**, in the reader's371 language, not what the code asserts. Someone who has never read the spec372 must be able to follow the report.3733. **Every step's evidence comes from the helper.** It waits for the page to374 settle, screenshots, and files that step's console/network noise. Do not375 hand-roll `page.screenshot("01-...")` as primary evidence.3764. **`journey.scenario({ goal, preconditions })` is required**, at the top of377 the test. It supplies the report's goal and preconditions; without it the378 report falls back to the English E2E-META `Purpose`/`Environment`, which is379 contract text rather than an explanation for a reader.3805. **A report is written for every outcome** — passed, failed, and blocked —381 because the fixture tears the reporter down. Steps that already ran stay in382 the report when a later step fails, and the failing step keeps its own383 screenshot plus an error summary.3846. **Do not weaken an assertion to make a report green.** A journey that385 documents a known product gap stays FAIL and says so in its preconditions.386387### What a run produces388389`.e2e/journey-reports/<spec>/<test>/` (gitignored) receives:390391| File | Content |392|---|---|393| `report.md` | Result, commit SHA under test (and whether the tree was dirty), spec file, group/tags, start/end/duration, scenario goal, preconditions, gate reason when blocked, the step table, and per-step detail |394| `report.html` | The same content, self-contained: inline CSS only, screenshots by relative path, opens from the filesystem with no network |395| `NN-<step>.png` | One screenshot per step, in step order; a step that failed or was blocked is suffixed accordingly |396397Per-step "key logs" are browser `console` errors/warnings, page errors, failed398requests, and any response with status ≥ 400 — recorded as method, URL, and399status. Bodies are never captured. Every recorded string passes through400redaction: known credential variables, `Bearer` values, `token=`/`api_key=`401query parameters, the repository root, and the home directory are replaced. Keep402it that way; reports get published.403404Both files are also attached to the Playwright HTML report, so a CI run carries405them without a separate copy step.406407Set `E2E_JOURNEY_REPORTS` to redirect the output root. Never commit reports,408screenshots, or `.e2e/`.409410### Shortest complete example411412```ts413import { expect } from "@playwright/test";414415import { test } from "./helpers/e2e.ts";416import { createProjectAndSession, openProjectSession, scriptedModel,417 sendUserMessage, waitForRunTerminal, expandToolStep, cleanupJourney } from "./helpers/journeys.ts";418419/**420 * E2E-META421 * ... all eleven fields ...422 */423test("J9 用户可以拿到一次可核对的执行结果", { tag: "@mocked" }, async ({ journey, page }) => {424 journey.scenario({425 goal: "一位研究员要确认产品真的在本机执行了他要求的计算,而不只是在聊天。",426 preconditions: ["隔离栈已启动", "模型由旅程自带的本地 stub 驱动,不访问外部服务"],427 });428429 const marker = `J9-${Date.now()}`;430 const stub = await scriptedModel([431 { arguments: { command: `echo ${marker}` }, tool: "run_shell" },432 { text: "The requested check completed." },433 ]);434 const fixture = await createProjectAndSession(page, {435 model: { apiToken: stub.apiToken, baseUrl: stub.baseUrl, model: stub.model, name: `J9 ${Date.now()}` },436 projectName: `J9 ${Date.now()}`,437 sessionTitle: `J9 ${Date.now()}`,438 });439440 try {441 await journey.step(442 "进入会话并提出请求",443 "任务发出后运行走到完成态,主按钮回到可再次运行的状态。",444 async () => {445 await openProjectSession(page, fixture);446 const run = await sendUserMessage(page, fixture.session.id, "Check the workspace and report what you find.");447 expect((await waitForRunTerminal(page, fixture.session.id, run.id)).status).toBe("completed");448 },449 );450451 await journey.step(452 "展开工具步骤核对输出",453 "工具步骤展开后能看到这次执行的真实输出,说明命令确实跑过。",454 async () => {455 await expect(await expandToolStep(page, { contains: marker })).toContainText(marker);456 },457 );458 } finally {459 await cleanupJourney(page, fixture);460 await stub.stop();461 }462});463```464465A `@real` journey follows the same shape; put `requireRealEnv(...)` and466`await requireRealStack(...)` inside the **first** `journey.step`, so a run467without credentials still produces a report whose first step reads468`⛔ 前置未满足` with the gate reason.469470### Migration pace for existing specs471472Specs quarantined in `LEGACY` may adopt this incrementally; they are not473blocked on it. **A new file has no grace period**: any spec added as474`journey-*.spec.ts` complies on its first commit, and a non-journey spec that475grows into a full user flow should be renamed and converted rather than kept as476an untracked exception.477478## Browser mocked rules479480- The model is always a spec-owned local stub (`node:http` `createServer` on481 `127.0.0.1`, registered through `/api/models`) or a seeded fake such as the482 hang/slow models. Never read `E2E_LLM_*` in a mocked spec, and never rely on483 a real model configured in the stack's `.env`.484- Import `test` from `test/helpers/e2e.ts`, never directly from485 `@playwright/test`. Its automatic fixture aborts non-local HTTP(S),486 policy-closes non-local WebSockets, and forwards localhost/loopback traffic487 before any `beforeEach`, navigation, or request. `check-e2e-meta.mjs`488 enforces this import for every `Type: mocked` file, including fixme tests489 when later enabled.490- Keep `test/e2e-network-guard.spec.ts` passing as the request-level proof for491 blocked HTTPS/WebSocket and allowed local HTTP/WebSocket behavior.492- Backend egress cannot be intercepted from the browser; it stays local493 because the only model the spec registers is its own stub. When a mocked494 spec must prove a call happened, assert on the stub (request counters,495 captured bodies), not on timing.496- Mocked specs must pass repeatedly on a clean stack with no credentials.497498## Browser real rules499500- Real specs are explicit opt-in (`E2E_REAL=1`), tagged `@real`, and declare501 their external behavior and cost in E2E-META.502- At the start of **each real test body**, before `beforeEach`-equivalent503 navigation, API setup, LLM, search, paper-source, or MCP actions, call504 `requireRealEnv(testInfo, "E2E_LLM_BASE_URL", "E2E_LLM_MODEL", "E2E_LLM_TOKEN")`505 and `await requireRealStack(testInfo)`. Do not put the gate only at file506 scope. `check-e2e-meta.mjs` enforces both calls per real test body and their507 order relative to the first external action.508- If a real test genuinely needs no standard LLM credentials, call509 `allowRealEnvException(testInfo, "specific reviewed reason")`; the stack510 gate remains mandatory and the exception appears in test annotations.511- For seeded runtime state (for example, model registry or PubMed connector),512 call `testInfo.skip(true, "BLOCKED: ...")` with a specific reason before first use only513 when that identifiable precondition is absent. Once gates pass, product514 assertion failures must remain FAIL. Report all gate skips as BLOCKED, never515 as passes.516- Expect nondeterminism: assert user-visible invariants, not exact model517 output; use generous observable-condition waits, not fixed sleeps.518- Record what the run actually consumed (model, endpoints, connectors) and519 keep traces/screenshots so results can be reviewed without re-spending.520521## Writing browser specs522523- Organize files and `describe` blocks by user goal, not by implementation524 module. A journey may use shell, Python, an environment, a subagent, and525 artifact versioning in one coherent flow. Small negative/cancel/recovery526 cases may sit beside the journey they qualify. Name main-flow files after527 the goal (`journey-first-run.spec.ts`, `journey-deliver-result.spec.ts`),528 never after an internal tool (`shell.spec.ts`, `python.spec.ts`).529- Use `test/helpers/journeys.ts` for common user actions: model registration530 and selection, Project/Session setup, natural-language submission, Run531 terminal-state waiting, permission handling, timeline/tool-process reading,532 environment revision lookup, and opening the environment or artifact533 surfaces. Use `scriptedModel(mainSteps, subagentSteps?)` for deterministic534 tool/text sequences; it routes general-purpose subagent requests by their535 preset system marker. Use `expandToolStep(page, { contains })` with a536 journey marker instead of making `run_shell` or `run_python` the only537 locator. `artifactTree(page)` deliberately exposes the declared catalog,538 `@` candidates, and the opt-in physical workspace tree as separate views.539 The helpers return records, locators, and visible text; the spec still owns540 goal-specific assertions.541- Use the `journey` fixture from `test/helpers/e2e.ts` (implemented in542 `test/helpers/journey-report.ts`) to structure the test as user steps and543 produce its report. See [Journey steps and automatic544 reports](#journey-steps-and-automatic-reports-mandatory) for the full545 contract and a copyable skeleton.546547- Prefer semantic locators (`getByRole`, `getByLabel`, `getByText`); when only548 a CSS hook works, that is also an accessibility signal worth noting.549- Wait on observable conditions (toast visible, button state, request550 finished); never sleep-and-hope. Assert user-visible outcomes, not merely551 that a request or click happened.552- Create test data with unique names (`Date.now()` suffix) and clean up in553 `finally` where practical; leftover data must stay in the run's own data554 directory.555- Long-term regression specs live in `test/`; throwaway diagnostic specs stay556 in the E2E worktree and are never committed.557558## Browser screenshots, evidence, artifacts559560- **Journey specs take no screenshots of their own.** `journey.step()` names561 them in step order, waits for the page to settle, and files them with the562 step they belong to. Add evidence by adding or resplitting a step, never by563 calling `page.screenshot()` — `check-e2e-meta.mjs` rejects that.564- Non-journey specs that still need an ad-hoc shot write to `screenshots/`565 relative to the Playwright cwd (i.e. `.e2e/screenshots/`), or to566 `E2E_SCREENSHOTS` when the spec supports it, named in step order.567- Shoot key success and failure states only after the target text/state is568 stable; avoid skeletons, animation remnants, and clipped controls. Journey569 screenshots already pass Playwright `animations: "disabled"` (and570 `caret: "hide"`) from `journey.step()`. Ad-hoc `page.screenshot()` calls571 must set the same `animations: "disabled"` option; omitting it leaves CSS572 transition ghosts in otherwise stable shots. Eyeball every screenshot after573 the run; retake any that does not match its caption — a screenshot that574 contradicts its step description is a report defect.575- Failures keep the automatic failure screenshot and trace576 (`trace: retain-on-failure`); reports land in `playwright-report/` and577 `test-results/` (including `results.json`) next to the config in use. Journey578 reports are additionally attached to each test in the Playwright HTML report.579- A run report references evidence by relative path inside the run's artifact580 directory and states: SHA under test, worktree, ports, config baseline,581 start and test commands, health status, per-case expected/actual, and582 failure attribution. Report discovered, executed, passed, failed, and583 skipped/fixme counts separately; discovered or skipped tests are not passes.584- Never commit `.e2e/` (which contains `journey-reports/`),585 `test/node_modules`, `test/playwright-report/`, `test/test-results/`,586 screenshots, traces, or logs — all gitignored. Committed files are the specs,587 `test/e2e.package.json` + `test/e2e.package-lock.json`,588 `test/playwright.config.ts`, `test/helpers/` (including589 `helpers/journey-report.ts`), `test/sync-e2e.mjs`, and590 `test/check-e2e-meta.mjs`.591- To hand a report to a reviewer, copy `report.md`, `report.html`, and that592 directory's `.png` files out as a unit — the HTML references the images by593 relative name, so the folder must stay together.594595## Failure attribution596597Prefer negative and fault injections that do not modify tracked files. Keep598the target worktree read-only. If tracked changes are unavoidable, use a599separate worktree derived from the target and create a test-only commit or an600exact index/patch snapshot before injection. Restore from that snapshot and601report the derived commit when present; never pollute the target candidate.602603| Type | Judgment |604|---|---|605| Product defect | UI/API/Gateway/Runner/orchestration breaks the user goal or gives no reasonable feedback |606| External environment | A third-party dependency failed and the product prompted or degraded correctly |607| Test defect | Selector, wait, timeout, fixture, or assertion is wrong; the product behaves |608| Test infrastructure | Worktree/port/data clash, wrong service, build ≠ SHA, dependency assembly or discovery error |609610A timeout is a tripwire, not a conclusion: confirm whether the run was issued,611the service responded, and the UI/API/CLI gave feedback. Every verdict needs at least612one piece of evidence (network log, console, service log, product message,613screenshot, trace); timeouts need two points along that chain. An external614failure the product swallows silently is still a product defect. Fix test or615infra problems and rerun; record `BLOCKED` when the cause cannot be separated.616617## Cleanup618619Archive evidence first, then stop the stack, free the ports, and remove run620data and `.e2e/` from the assigned worktree. Do not remove the task worktree;621its lifecycle belongs to the task owner. Remove a separate detached E2E622worktree only when this run created it for one of the justified isolation623reasons above. The main worktree must remain untouched.