Kata Code E2e Testing

E2E test author

Before writing code

1. Read e2e/README.md and the relevant product spec (docs/specs/2026-06-21-e2e-testing-foundation-design.md).
2. Inspect existing tests under e2e/tests/ and reusable blocks in e2e/src/harness/ and e2e/src/flows/.
3. Explore the UI on the real harness first, then codify locators in flows/ — do not guess selectors from product code alone.

How to explore a new flow (preferred)

Default: Playwright UI / debug on the spec, not playwright-cli.

Electron E2E runs through an isolated dev stack (Vite + Playwright-launched Electron). Playwright UI/debug uses the same launch, pairing, auth, and ports as the real test. playwright-cli attach --cdp=… is optional and brittle here (port conflicts, short-lived test processes).

# Pick the smallest existing spec closest to your flow, then:
vp run e2e:ui --project desktop-dev --grep '@settings'

# Step-through with inspector:
PWDEBUG=1 vp run e2e:headed --project desktop-dev --grep '@agent'

Workflow:

1. Run UI/debug on the nearest starter spec (@smoke, @settings, @agent).
2. Walk the user-visible steps; note stable locators (role, label, deliberate data-* contracts).
3. Add or extend a helper in e2e/src/flows/ — keep specs thin.
4. Re-run headed on the tag; confirm pass before moving to the next spec.

Roll out starter specs one at a time in headed mode; give the maintainer the exact verify command after each passes.

Optional CDP (only if you need playwright-cli against a running Electron):

KATACODE_DESKTOP_REMOTE_DEBUGGING_PORT=9333 vp run e2e:headed --project desktop-dev --grep '@smoke'
# use a free port; 9222 is often taken

The harness forwards KATACODE_DESKTOP_REMOTE_DEBUGGING_PORT as --remote-debugging-port on the raw Electron launch.

Rules

• Compose tests from e2e/src/harness/ and e2e/src/flows/ — do not duplicate launch, auth, isolation, or navigation logic in spec files.
• Keep generic Electron/process concerns in harness/ and Kata UI/product language in flows/.
• Do not mock application services: no Playwright route().fulfill(), HAR replay, MSW, or fake provider backends.
• Store secrets and auth state only under ignored paths (e2e/.auth/, Playwright output dirs, local .env.local).
• Tag every spec with at least one feature tag, e.g., @smoke, @auth, @settings, @agent.
• For new features, create new tags.
• Fail loudly with the missing env var or prerequisite when credentials are absent — never skip assertions silently.
• Prefer user-visible locators (role, label, text). Add data-testid in product code only when no durable accessible locator exists and the attribute is a deliberate test contract.
• Default to one worker for authenticated mutable flows unless additional isolated test accounts exist.
• Use reasonable timeouts from e2e/src/config/timeouts.ts — tests should fail fast, not hang.

Architecture (do not fight this)

Do not run dev:desktop inside E2E — it would spawn a second Electron and cause EPIPE / duplicate backends.

Release (desktop-release)

• Packaged apps load the renderer from the embedded server (http://127.0.0.1:<KATACODE_PORT>/), not Vite.
• Do not pass VITE_DEV_SERVER_URL — inherited dev env makes the app enter dev mode, open DevTools, and show a blank window trying to reach a Vite port that is not running. launchEnv.ts strips dev-only vars for release (same rule as apps/desktop/scripts/start-electron.mjs).
• Renderer window detection uses serverPort, not webPort. No dev:web stack is started for release.
• vp run e2e:release is headed on macOS — Playwright Electron opens a visible packaged app window; no need to add e2e:headed.

Known Electron / UI gotchas

Auth

• Do not drive Google OAuth in the Electron page. Desktop OAuth opens an external browser via desktopBridge; in-page email/password locators will not work.
• Use @clerk/testing **clerk.signIn({ page, emailAddress })** (ticket flow) in signInWithClerkGoogleTestUser. Requires CLERK_SECRET_KEY + a Clerk user for KATACODE_E2E_GOOGLE_EMAIL.
• Clerk loads on the app shell once cloud config is present; wait for command-palette-trigger, then clerk.loaded.
• Sign in to Kata Code Connect appears only in the Settings sidebar footer, not on the main chat sidebar.

Navigation

• Electron uses hash history (#/settings/general). Bare page.goto("/settings/…") is invalid.
• Prefer in-app clicks (openSettings in settings.ts clicks the sidebar Settings data-sidebar="menu-button").
• Settings title on Electron is a <span>, not a heading — wait on a panel control (e.g. getByLabel("Theme preference")).

Toasts and overlays

• Provider update toasts can block clicks on settings controls and model picker options.
• Call dismissBlockingToasts(page) (toast close: [data-slot="toast-close"]) before theme or model selection when needed.

Agent / model picker

• Before sendAgentInstruction, call **selectComposerModel(page, turn.model)** so the test uses KATACODE_E2E_AGENT_MODEL (e.g. gpt-5.4-mini), not whatever default the composer had.
• Flow: click [data-chat-provider-model-picker="true"] → fill Search models… with slug tokens (hyphens → spaces) → click matching [role="option"].
• @agent tests need KATACODE_E2E_AGENT_PROVIDER, KATACODE_E2E_AGENT_MODEL, and the matching provider API key.

Reusable building blocks

Harness (e2e/src/harness/)

• testFixtures.ts — test, appWindow, authenticatedAppWindow, runContext, launchTarget
• appLaunch.ts — dev stack + Electron launch (technical readiness only)
• isolatedRun.ts — temp KATACODE_HOME, shared dev-runner port allocation, cleanup
• readiness.ts — TCP / Vite readiness probes
• env.ts — prerequisite checks (readClerkPrerequisites, readAgentProviderPrerequisites, …)

Flows (e2e/src/flows/)

Assertions (e2e/src/assertions/)

Launch health checks only (assertNoFatalLaunchErrors). Import flows directly for actions and UI waits.

Typical test shapes

Smoke (@smoke) — no auth

import { assertNoFatalLaunchErrors } from "../../src/assertions/appAssertions.ts";
import { E2E_TAGS } from "../../src/config/tags.ts";
import { test, expect } from "../../src/harness/testFixtures.ts";

test.describe(`App launch ${E2E_TAGS.smoke}`, () => {
  test("launches Electron past pairing and reaches the app shell", async ({
    launchedApp,
    appWindow,
  }) => {
    await expect(appWindow.getByTestId("command-palette-trigger")).toBeVisible();
    assertNoFatalLaunchErrors(launchedApp.readFatalErrors());
  });
});

Settings (@settings) — Clerk ticket sign-in + theme

import { E2E_TAGS } from "../../src/config/tags.ts";
import { expectResolvedTheme, openSettings, setTheme } from "../../src/flows/settings.ts";
import { test } from "../../src/harness/testFixtures.ts";

test.describe(`Settings theme ${E2E_TAGS.settings}`, () => {
  test("persists dark theme after reload", async ({ authenticatedAppWindow }) => {
    await openSettings(authenticatedAppWindow);
    await setTheme(authenticatedAppWindow, "dark");
    await authenticatedAppWindow.reload();
    await openSettings(authenticatedAppWindow);
    await expectResolvedTheme(authenticatedAppWindow, "dark");
  });
});

Agent (@agent) — model selection + real LLM reply

import {
  assertAgentPrerequisites,
  expectAssistantReply,
  selectComposerModel,
  sendAgentInstruction,
} from "../../src/flows/agentChat.ts";
import { createOrOpenProject, createSeededWorkspace } from "../../src/flows/workspace.ts";
import { E2E_TAGS } from "../../src/config/tags.ts";
import { E2E_TIMEOUTS } from "../../src/config/timeouts.ts";
import { test } from "../../src/harness/testFixtures.ts";

test.describe(`Deterministic agent chat ${E2E_TAGS.agent}`, () => {
  test.describe.configure({ timeout: E2E_TIMEOUTS.agentTestMs });

  test("returns the exact expected assistant message from a real provider", async ({
    authenticatedAppWindow,
    runContext,
  }) => {
    const turn = assertAgentPrerequisites("deterministic agent chat");
    const seededPath = await createSeededWorkspace(runContext, "agent-chat-basic");
    await createOrOpenProject(authenticatedAppWindow, seededPath);
    await selectComposerModel(authenticatedAppWindow, turn.model);
    await sendAgentInstruction(authenticatedAppWindow, turn.prompt);
    await expectAssistantReply(authenticatedAppWindow, turn.expected, turn);
  });
});

Verification commands

vp run e2e --list --grep @your-tag
vp run e2e:headed --project desktop-dev --grep @your-tag   # maintainer verify
vp run e2e:ui --project desktop-dev --grep @your-tag       # explore / debug
vp check
vp run typecheck
vp test e2e/src/**/*.test.ts

Suggested rollout order for the starter suite: @smoke → @settings → @agent.

For release-only coverage (headed on macOS — no e2e:headed needed):

KATACODE_E2E_RELEASE_APP="/path/to/Kata Code.app" vp run e2e:release --grep @smoke