E2e Testing

ORGII E2E Testing

ORGII keeps two separate E2E surfaces. Do not use one as proof for the other.

1. Core UI E2E — tests/e2e/specs/core/, driven by WebDriverIO against the debug-built Tauri app.
2. Rust runtime E2E — src-tauri/crates/e2e-test/, a Rust HTTP client against debug-only /agent/test/* endpoints.

A Rust HTTP scenario can prove runtime state. It does not prove a user can see, click, or recover through the rendered UI. If a feature has a button, card, menu item, wizard field, status pill, or visible chat behavior, it needs rendered UI coverage.

Core UI E2E policy

tests/e2e is the final UI regression suite. Keep it small and clean.

Rules:

• Extend an existing core spec before creating a new one.
• Do not put historical audits, migration sweeps, subsystem experiments, or one-off debug specs under tests/e2e.
• Every UI spec must perform a real rendered action or assert a real rendered result. Debug helpers may seed state, but cannot be the only proof.
• Provider capacity failures, especially Gemini 429/rate-limit/capacity errors, are infra/provider issues unless ORGII mishandles the rendered error or runtime state.
• OAuth refresh failures with permanent invalid-token messages are account health blockers. ORGII should record the failure and disable the account immediately; UI E2E should report them separately from product regressions and continue through configured account/model fallback chains when available.

Anti-false-prosperity policy

A green E2E result is not accepted unless it proves the production behavior being claimed.

• Do not mark a scenario PASS when the critical action is replaced by a frontend mock, synthetic success flag, debug-only responder, or helper that bypasses the production command/event path.
• Debug helpers may establish deterministic preconditions, but the user-visible action under test must still use the production click/command/dispatcher path.
• Do not use corrective follow-up prompts, extra retry prompts, or stronger second instructions to make an agent pass after the original user path failed. The first-path failure is the product signal.
• Do not count a matrix run as proof for multiple labels unless each requested label produced independent evidence. Combined fallback output is not per-label proof.
• Do not promote old green rows after prompt text, harness setup, account fallback, or product semantics changed. Rerun only the affected rows and record current-code evidence.
• For interactive cards, assert the full lifecycle: rendered reason/body, actionable button, production response command, backend/runtime state change, and final rendered state. A pill text change alone is not enough.
• For mode/tool claims, assert session-scoped effective tools (agent_list_effective_tools_for_session, /agent/test/effective-tools/:session_id, or __e2e.listEffectiveToolsForSession) rather than global registry or historical renderability.
• Treat provider quota/capacity blocks as BLOCKED, not PASS; never route around them silently to manufacture green coverage.

Real-interaction regression policy

If a bug was found by a human using the rendered app, the regression test must replay the human interaction path closely enough to fail before the fix.

• Prefer browser.keys, real clicks, focus/blur, menu navigation, and visible-state waits over browser.execute text injection. Direct DOM mutation is allowed only for deterministic setup, never as proof that input handling works.
• Contenteditable tests must first prove the keystroke/input actually changed the rendered editor text before asserting menus or buttons. A keydown-only signal (menu opened but editor.textContent stayed empty) is a harness/product bug signal, not proof that @query or /query works.
• Prefer real keyboard clearing for contenteditable surfaces (Cmd/Ctrl+A, delete/backspace, then browser.keys(...)) and wait for the rendered editor text to stabilize before menu assertions. Do not require document.activeElement to remain the editor after @ or / opens a portal/menu; focus may legitimately move while the editor text remains the source of truth.
• If Tauri WebDriver element-click/focus is flaky for a contenteditable surface, the spec may use document.execCommand('insertText') plus a real bubbling InputEvent to exercise the product onInput path, but it must assert the editor text, query consumption, and final visible result. Do not use plain textContent = ... as the behavior under test.
• If a test helper replaces DOM text directly, it must be limited to deterministic seed/setup. A regression for inline @/slash/menu behavior must use keyboard input or an InputEvent path that can fail when React draft state restores stale text.
• A composer/menu test must assert all user-visible invariants: previous draft text preserved, transient query text consumed, inserted pill/chip visible, focus restored when product requires it, and send/stop state correct.
• Stop/Pause/Queue tests must assert immediate button state, composer interactivity, stream cessation, queue retention/non-autoflush, and draft restoration when a not-yet-sent message is canceled.
• Use seeded events only to create durable transcript preconditions. Rendering assertions must still inspect the actual chat UI, including grouped/aggregated blocks, not only data-testid fragments that disappear under aggregation.
• A test helper that calls setTextarea, insertText, ensureRepoSelected, or a debug seed path must include a comment or assertion explaining which production behavior is still being exercised afterward.
• If a prior test used a shortcut and missed a bug, update the skill/spec so future tests forbid the shortcut for that class of interaction.

Multi-repo workspace regression policy

Multi-root behavior must be treated as a first-class product contract, not a display patch.

• Distinguish durable session root, primary workspace folder, active editor folder, search result source repo, and tool-event target path. Do not use active editor focus as a durable session root unless the user explicitly selected it.
• Explicitly test that activeFolderAtom / active editor focus can move to a secondary repo without changing the durable launch root. Agent session launch defaults to the primary workspace folder; active folder is only a UI/current-focus concept.
• Every multi-repo UI test should include at least two repos with colliding filenames so source attribution cannot be inferred from basename alone.
• @ search and context menus must render persistent source evidence (repoName/path badge), not only hover-only titles.
• File-path extraction must go through a shared extractor that handles canonical payload variants (file_path, filePath, target_file, targetFile, path) across backend normalization, frontend props, summaries, and grouped chat blocks. Do not add component-local a || b || c chains.
• Read-file rendering tests must cover single blocks, grouped ReadFileGroup, and aggregate ActionSummaryGroup summaries, including camelCase Cursor-style tool payloads.
• Multi-repo session launch tests must assert the selected durable repo path in the launch payload/runtime snapshot, and separately assert that UI search/source badges remain accurate for non-primary repos.
• Multi-root E2E setup helpers must not silently call single-repo pinning (repoPath: E2E_REPO_PATH, ensureRepoSelected, or equivalent) after seeding multiple folders. If a creator/helper needs account/model setup only, pass through the existing selected multi-root workspace and assert workspaceFolders/source evidence afterward.
• Multi-repo path-rendering tests must use self-contained fixture paths with colliding basenames and payload key variants (targetFile, file_path, nested success.filePath, etc.). Do not depend on another local checkout such as claude_code, and do not accept generic labels like file as path evidence.
• Multi-repo search tests must validate both the visible source badge and the selected path/pill value. A menu that merely contains two basenames is not enough; the chosen secondary repo result must survive click/keyboard selection into the composer context.
• When a multi-repo bug is fixed in one surface, sweep all equivalent surfaces: session creator, existing chat composer, context menu, event normalizer, props extraction, tool-call summary, grouped transcript rendering, and E2E seed helpers.
• Audit duplicate workspace state sources before adding patches. If both a canonical store path and an older/legacy workspace atom module exist, tests must import the production path and the diff must not add another derived source of truth.
• Any E2E helper that sets activeFolder, selectedRepo, workspaceFolders, or launch workspace fields must return a snapshot of all related atoms/paths and the test must assert the durable/active distinction immediately. If a failure message shows the target path inside the folder dump but matching failed, inspect argument marshaling and path normalization before adding fallback display logic.
• Do not fix multi-repo bugs with display-only band-aids. A valid fix names the data contract, centralizes extraction/resolution once, wires all consumers to that contract, and adds negative tests that would fail if a component-local fallback or single-repo pinning returned.

Matrix evidence policy

For requested provider/runtime matrices, each row needs current-code evidence and a clear outcome.

• Record the exact account/model/runtime row, command/spec/scenario, and result (PASS, BLOCKED, or FAIL).
• A fallback due to Gemini 429/capacity may satisfy the user flow only if the row records the original provider block and the fallback model that actually produced evidence.
• Do not claim “9 matrix all green” from a subset run, a prior commit, or a combined fallback. Every row must produce independent evidence or be explicitly marked BLOCKED with provider/account reason.
• Matrix rows should reuse deterministic fake-provider/debug bridges for product invariants and reserve live-provider rows for integration smoke, otherwise provider flakiness hides product regressions.

Workspace fixture policy

Core UI E2E must not depend on yorg_frontend, yoyo-evolve, or any external local project.

The WDIO runner creates a self-contained git fixture repo by default:

• Path: /tmp/orgii-e2e-workspace-repo
• Rebuilt at runner startup
• Contains README.md, package.json, src/math.ts, and an initial git commit
• Safe for agent mutation tests

Only set E2E_REPO_PATH when intentionally overriding with another sandbox git repo. The runner must reject explicit paths that do not exist, are not git repos, or lack the baseline files.

Session launch specs should pass the fixture repoPath through the same session configure/launch caller path the user uses. Do not add a separate before hook that only calls ensureRepoSelected; that helper can time out before the app is fully settled and can mask the real launch path with WebDriver harness failures.

Recommended isolated UI run when the developer app may already be using 1998:

E2E_ISOLATED_RUN=1 \
E2E_ORGII_HOME="/tmp/orgii-e2e-home" \
E2E_FRONTEND_PORT=21998 \
E2E_WEBDRIVER_PORT=24444 \
E2E_IDE_SERVER_PORT=23847 \
npm test

WDIO managed runs must not kill or reuse a developer's active ORGII app by default. The runner should fail fast if its managed ports are occupied unless E2E_ALLOW_PORT_CLEANUP=1 is explicitly set. When E2E_FRONTEND_PORT differs from 1998, the WDIO runner must make that real by building the webdriver debug app against a temporary Tauri devUrl pointing at the requested port, then restoring src-tauri/tauri.conf.json exactly. Merely starting webpack on a non-1998 port is false isolation because an unpatched debug app still loads http://localhost:1998.

Rust runtime E2E policy

e2e-test is a deterministic runtime contract suite, not a second UI suite and not a live-provider platform matrix. Keep it much smaller than the historical audit-era suite.

Keep:

• Backend/runtime invariants not covered by rendered UI E2E.
• Deterministic debug-endpoint coverage for memory, learning, permissions, worktree, session recovery, housekeeping, LSP, gateway/sync/MCP contracts, subagent dispatch, and tool execution invariants.
• Tool-policy and agent-definition contracts that are hard to observe from UI alone, especially positive/negative schema or policy assertions. Use the session-scoped effective-tools surface (agent_list_effective_tools_for_session, /agent/test/effective-tools/:session_id, or __e2e.listEffectiveToolsForSession) rather than global list_all_tools or registry-only /agent/test/tool-schemas/:session_id when asserting what a running agent can actually see in a mode-filtered prompt.
• Scenarios with stable setup, stable assertions, and explicit teardown/isolation.

Delete or move out:

• Historical phase/audit scenarios whose invariant is already covered by a canonical scenario.
• Long-running live-LLM scenarios that mainly duplicate UI/platform matrix behavior.
• Provider-specific smoke tests that are better covered by core UI matrix rows.
• Memory/learning tests that only prove the model can recall rendered text; keep state/DB/policy pins instead.
• Plan lifecycle tests that assert user-visible card/button behavior; keep only backend policy/snapshot invariants in Rust.
• Scenarios whose only assertion is HTTP 200 or loose text without a stable invariant.
• Dead helper modules/functions not registered in main.rs and not called by a registered scenario.

When cleaning Rust E2E:

1. Inspect src-tauri/crates/e2e-test/src/main.rs scenario registry.
2. Count groups with cargo run -p e2e-test -- --list or a local registry parser.
3. Remove entries only when their invariant is duplicated, obsolete, flaky by design, or moved to UI E2E.
4. Delete the module/function after removing the registry entry.
5. Run cargo check -p e2e-test and cargo fmt.

Choosing the right layer

Commands

Rust runtime:

cd src-tauri
cargo run -p e2e-test -- --list
cargo run -p e2e-test -- --scenario plan-mode-denies-writes
cargo run -p e2e-test -- --group memory
cargo check -p e2e-test
cargo fmt -p e2e-test

Core UI:

cd tests/e2e
# Full matrix (requires E2E_ALLOW_PORT_CLEANUP=1 if dev app is running)
E2E_ALLOW_PORT_CLEANUP=1 npm test -- --spec './specs/core/session-controls-ui.spec.mjs'

# Single scenario
E2E_ALLOW_PORT_CLEANUP=1 E2E_CONTROL_SCENARIOS=rewind npm test -- --spec './specs/core/session-controls-ui.spec.mjs'
E2E_ALLOW_PORT_CLEANUP=1 E2E_CONTROL_SCENARIOS=plan-build-direct npm test -- --spec './specs/core/session-controls-ui.spec.mjs'
E2E_ALLOW_PORT_CLEANUP=1 E2E_CONTROL_SCENARIOS=plan-update npm test -- --spec './specs/core/session-controls-ui.spec.mjs'
E2E_ALLOW_PORT_CLEANUP=1 E2E_CONTROL_SCENARIOS=plan-edit-resend npm test -- --spec './specs/core/session-controls-ui.spec.mjs'

# Core UI:
E2E_ALLOW_PORT_CLEANUP=1 npm test -- --spec './specs/core/chat-rendering-ui.spec.mjs'

Result-driven orchestration regressions

Rendered orchestration tests must start from the final user result and durable end-state, not from implementation breadcrumbs. A test that creates rows/cards but would still pass when a run stops with incomplete work is a false positive.

For Agent Org / multi-member / queue scenarios, every complex spec should state or encode:

• Final user outcome: what the org/team should have achieved.
• Final DB invariants: run status converged; all completed tasks are actually completed; open work is visibly blocked/abandoned; no in_progress task lacks an owner.
• Final UI evidence: resident member sessions are visible and switchable in the left sidebar, member transcripts open, and task board status does not contradict run/session state.
• Runtime path evidence: production launch, task tools, member wake/drain, inbox delivery, and member-session messaging paths ran; debug helpers only seed or inspect.
• Anti-false-positive checks: scenario-named tasks, passive inbox rows, synthetic cards, or a second corrective prompt do not count as success.
• Latest-session evidence: after a user reports a stuck or contradictory Agent Org run, inspect the newest run/session/task/inbox durable state and the latest terminal/app log before claiming the fix is verified. Do not infer success from an older green scenario or from a different synthetic run.

A Rust runtime E2E that posts protocol messages or calls debug endpoints is not proof that Agent Org works in the app. Rendered Agent Org acceptance must drive the production launch path from the UI, wait for production wake/drain/member-session turns, and then assert both UI and durable DB finality. Unit tests and Rust E2E can pin regressions, but they cannot be used as the sole evidence for “Agent Org advances correctly.”

Minimum failure cases that a valid orchestration spec must catch:

• A running run with no active member session and no unread inbox work to wake/drain (stalled run).
• status = "in_progress" with owner = null (ownerless work persisted).
• A completed run that still has pending or in_progress tasks.
• A run that appears visually populated but cannot make forward progress without a corrective second prompt.

• A running run with pending / in_progress tasks, no active member session, and no unread inbox work to wake/drain.
• A ready assigned pending task whose dependencies are all completed, but whose owner has no unread TaskAssigned inbox row and no active member turn.
• Unread org inbox rows that remain unread after the owner/member production session has gone idle/completed a turn.
• status = "in_progress" with owner = null, or status = "in_progress" set by the coordinator for another member rather than by the owning member's claim/drain path.
• A completed run that still has pending or in_progress tasks.
• Member sessions visible in the coordinator overview but absent from the left sidebar.
• Multiple org members sharing the same agent_id / agent_definition_id while inbox delivery, wake, drain, task owner, and task-tool authorization are only keyed by agent_id.
• A run that appears visually populated but cannot make forward progress from the original user prompt without a corrective second prompt.

File changes panel and diff view

The inline file-review panel was removed in fbf20c78. The composer "files pill" now opens Agent Station Diff view instead of expanding an inline card. Tests that assert file-change review must account for this:

• The files pill (data-testid="composer-section-files") click calls openAgentStationDiff, which sets chatPanelMaximized=false, stationMode="agent-station", simulatorSelectedAppAtom=AppType.DIFF, and replayModeAtom="replay".
• chatPanelMaximized must be false before ActivitySimulator (and thus SimulatorWorkstationTabHeader) renders. If the chat panel is maximized, the diff view pane is suppressed and its buttons are invisible.
• Undo All button: data-testid="file-changes-undo-all" in SimulatorWorkstationTabHeader (rendered only when pendingCount > 0).
• Redo All button: data-testid="file-changes-redo-all" in SimulatorWorkstationTabHeader (rendered only when redoSnapshotAnchors.length > 0).
• The E2E helper __e2e.openAgentStationDiff() sets the same atoms as the product pill callback and is available as a fallback when Tauri WebDriver element.click() misses React synthetic events. Use invokeE2E("openAgentStationDiff") after failed pill-click retries.
• waitForFileChangesPanel in agentQueuedWorkspaceHelpers.mjs encapsulates this logic: it retries pill click 3×, then falls back to invokeE2E("openAgentStationDiff"), then waits for [data-testid="file-changes-undo-all"] or [data-testid="replay-tab-diff-filter"] to appear.
• For plan-build-direct, always call waitForRuntimeIdle() before waitForFileChangesPanel() — the Undo All button only activates after the build turn is fully idle.

Plan, rewind, and streaming regressions

Rendered plan tests must pin the caller path, not only derived UI helpers:

• Rewind/edit-resend must invalidate stale queued turns and cancel the active turn before sending the replacement message.
• Plan update/edit-resend tests must assert no duplicate pending/drafting cards, only the latest plan is buildable, and stale revisions remain visible only as archived history when appropriate.
• Plan card diagnostics must distinguish surfaces by data-plan-surface: transcript cards in chat history, current cards in the pending review bar, and communication-side preview cards.
• Stop/Send button E2E clicks must be atomic with the expected data-state.
• Long-running debug HTTP endpoints must be called from the WDIO Node process, not through browser.executeAsyncScript(fetch(...)).
• Streaming marker assertions must wait for the full expected marker, not only for assistant text to become non-empty or change.

Hard rules

• Never change product business semantics to make an E2E pass. E2E fixes should wire the existing backend/runtime path, assert the current UI contract, or expose a real product bug.
• Never run mutation-capable UI E2E against yorg_frontend.
• Never claim “E2E passed” after only running unit tests, focused module tests, cargo check, or Rust protocol/debug-endpoint scenarios. Name the exact surface that passed: unit, Rust runtime E2E, or Core UI E2E.
• Never add a rendered UI claim to Rust-only coverage.
• Never add a debug endpoint that tests only a helper when the bug is in the caller path.
• Never preserve an obsolete scenario just because it once caught a phase bug; keep the invariant, not the phase artifact.

Turn lifecycle and queue E2E

The turn lifecycle is controlled by a FSM in src/engines/SessionCore/control/turnLifecycle.ts. Key concepts for E2E:

• turnPhase: "idle" | "dispatching" | "running" | "stopping" — use inspectChatState().turnPhase to assert turn state precisely.
• turnGeneration: monotonically increasing counter; each new turn gets a new generation. Stale terminal signals from old turns are ignored.
• runtimeStatus: derived from FSM + provider signals. Use waitForRuntimeIdle() to wait for runtimeStatus === "idle" and turnPhase === "idle".
• Queue tests must assert queuedMessages array contents, not just UI queue item count. Use inspectChatState().queuedMessages.
• After Stop, queued messages with requiresExplicitDispatch=true must not auto-flush. Assert queue retention before any follow-up send.

CLI session reload

CLI sessions (claude-code, codex, cursor-cli, gemini-cli) reload history from SQLite via cliAdapter.loadHistory after a browser refresh. Key rules:

• After cli_agent_truncate_after_chunk (edit-resend / rewind), the product code calls deleteCachedSession and evictSession to ensure a clean reload. Tests that reload after rewind must wait for chatEventCount > 0, not just activeSessionId match.
• reloadAndOpenActiveSession retries openSession up to 3× with 3s gaps — CLI adapter settling after reload is a known race that does not affect real users (who wait for UI to render before clicking).
• Do not mark CLI reload as a product failure if chatEventCount: 0 appears only after a programmatic browser.refresh() + immediate openSession. Verify with multiple runs before treating as a stable product bug.