Visual Test

Visual Test

Runs a real browser against the Zulip test server and takes screenshots you can read as images to verify layout, colors, positioning, text content, etc.

Steps

1. Write the puppeteer test script

Create web/e2e-tests/_claude_<feature>_test.test.ts using this template:

import type {Page} from "puppeteer";

import * as common from "./lib/common.ts";

async function visual_test(page: Page): Promise<void> {
    await common.log_in(page);
    await common.screenshot(page, "step-1-logged-in");

    // Navigate, interact, and screenshot each significant state.
    // See "Available helpers" below.
}

await common.run_test(visual_test);

Adapt the body to exercise whatever UI you need to verify. Take a screenshot at every visually significant state using descriptive names like step-2-color-picker-open, step-3-color-selected.

Important patterns:

These patterns are derived from the existing Puppeteer tests in web/e2e-tests/. Follow them to write reliable, non-flaky tests.

Waiting: never use hardcoded timeouts

The existing test suite has essentially zero setTimeout calls (the two in common.ts are explicitly commented workarounds for specific animation flakes). Always wait for the specific condition you expect instead. The three main waiting primitives, in order of preference:

• waitForSelector — wait for an element to appear or disappear. This is the most common pattern in the test suite (100+ uses):

  // Wait for element to be visible (most common)
  await page.waitForSelector("#left-sidebar", {visible: true});
  
  // Wait for element to disappear (e.g., overlay closed, row deleted)
  await page.waitForSelector("#subscription_overlay", {hidden: true});
  

• waitForFunction — wait for a condition that can't be expressed as a single selector (text content, element count, attribute value, application state):

  // Wait for specific text content
  await page.waitForFunction(
      () => document.querySelector(".save-button")?.textContent?.trim() === "Save changes",
  );
  
  // Wait for element count after filtering
  await page.waitForFunction(
      () => document.querySelectorAll(".linkifier_row").length === 4,
  );
  
  // Wait for an input's value to update
  await page.waitForFunction(
      () => document.querySelector<HTMLInputElement>("#full_name")?.value === "New name",
  );
  
  // Wait for focus to land on a specific element
  await page.waitForFunction(
      () => document.activeElement?.classList?.contains("search") === true,
  );
  
  // Wait for internal app state via zulip_test
  await page.waitForFunction(
      (content) => {
          const last_msg = zulip_test.current_msg_list?.last();
          return last_msg !== undefined && last_msg.raw_content === content
              && !last_msg.locally_echoed;
      },
      {},
      content,
  );
  

• waitForNavigation — only for actual full-page navigations (form submits, reloads). Wrap with Promise.all when the navigation is triggered by an action:

  await Promise.all([
      page.waitForNavigation(),
      page.$eval("form#login_form", (form) => { form.submit(); }),
  ]);
  

Interacting with elements

• page.click(selector) is the standard for clicking. When it's unreliable (overlapping elements, timing), fall back to clicking via evaluate — several existing tests do this with a comment explaining why:

  // When page.click() is unreliable, click via the DOM directly
  await page.evaluate(() => {
      document.querySelector<HTMLElement>(".dialog_submit_button")?.click();
  });
  

• page.type(selector, text) for typing. Use {delay: 100} when typing triggers a typeahead or filter that needs per-keystroke updates:

  await page.type('[name="user_list_filter"]', "ot", {delay: 100});
  

• common.clear_and_type(page, selector, text) to replace existing input content (triple-click + Delete + type).

• common.fill_form(page, selector, params) to fill multiple form fields at once — handles text inputs, checkboxes (by toggling), and <select> elements.

• common.select_item_via_typeahead(page, selector, str, item) to type into a field and pick a typeahead suggestion.

• page.keyboard.press("KeyC") for Zulip keyboard shortcuts. After pressing, wait for the resulting UI change:

  await page.keyboard.press("KeyC");
  await page.waitForSelector("#compose-textarea", {visible: true});
  

• Hover before clicking action buttons that only appear on hover (e.g., message action icons):

  const msg = (await page.$$(".message_row")).at(-1)!;
  await msg.hover();
  await page.waitForSelector(".message-actions-menu-button", {visible: true});
  await page.click(".message-actions-menu-button");
  

Navigating within the app

• Click sidebar items for in-app navigation:

  await page.click(".narrow-filter[data-stream-id='...'] .stream-name");
  await page.waitForSelector("#message_view_header .zulip-icon-hashtag", {visible: true});
  

• page.goto(url) for hash-route navigation:

  await page.goto(`http://zulip.zulipdev.com:9981/#channels/${stream_id}/Denmark`);
  

• common.manage_organization(page) to navigate to org settings, common.open_personal_menu(page) to open the personal menu.

Reading state with page.evaluate

Use page.evaluate() to read internal application state or DOM properties not accessible through selectors:

// Read internal Zulip state via the zulip_test global
const stream_id = await page.evaluate(
    () => zulip_test.get_sub("Verona")!.stream_id,
);

// Read DOM properties
const page_language = await page.evaluate(
    () => document.documentElement.lang,
);

Changing user settings via the API

Use page.evaluate with fetch() and reload, rather than clicking through the settings UI:

await page.evaluate(async () => {
    const csrfToken = document.querySelector<HTMLInputElement>(
        'input[name="csrfmiddlewaretoken"]',
    )?.value ?? "";
    await fetch("/json/settings", {
        method: "PATCH",
        headers: {
            "Content-Type": "application/x-www-form-urlencoded",
            "X-CSRFToken": csrfToken,
        },
        body: "user_list_style=1",
    });
});
await page.reload({waitUntil: "networkidle2"});

XPath selectors for text matching

When you need to match elements by text content, use XPath with common.has_class_x():

await page.waitForSelector(
    `xpath///*[${common.has_class_x("stream-name")} and normalize-space()="Verona"]`,
);

Assertions for visual tests

For visual test scripts, prefer a soft-assertion pattern that reports all failures rather than aborting on the first:

const results: string[] = [];
function check(name: string, ok: boolean): void {
    results.push(`${ok ? "PASS" : "FAIL"}: ${name}`);
    console.log(`${ok ? "PASS" : "FAIL"}: ${name}`);
}
// ... run checks ...
const failures = results.filter((r) => r.startsWith("FAIL"));
console.log(`\n${results.length - failures.length}/${results.length} tests passed`);

This keeps the test running through failures so you see all results, unlike assert which aborts on the first failure.

Verifying alignment

When using Puppeteer to verify visual alignment, do not rely on eyeballing screenshots — especially small full-page ones. Instead:

• Use page.evaluate() with getBoundingClientRect() to measure actual pixel positions of the elements you need aligned, and print them to the console. Compare the numbers.
• Always take both a full-page screenshot and a zoomed clip of the area of interest.
• For zoomed clips, calculate the clip region from non-fixed elements; fixed/sticky elements may report bounding-box positions that don't match their visual location on the page.
• Be aware that CSS nesting can scope styles to a specific parent (e.g., .parent .my-class) — reusing the same class name in a different context may not pick up the expected styles.
• To verify keyboard-focus styles, use real keyboard navigation (page.keyboard.press); programmatic .focus() doesn't reliably trigger :focus-visible and may be overridden by view-level focus management.
• Focus rings drawn as ::before / ::after pseudo-elements aren't visible in getComputedStyle of the focused element — verify them in a screenshot, not via computed styles.
• For visual changes, produce before/after screenshot pairs by writing one test and running it twice with a SCREENSHOT_SUFFIX env var (-old on main, -updated on your branch).

2. Run the test

./tools/test-js-with-puppeteer _claude_<feature>_test

The runner matches test file names by prefix, so you don't need the full filename or .test.ts suffix. This starts a fresh test server on port 9981, runs the script, and saves screenshots to var/puppeteer/. The test database is reset between test files.

On aarch64 (ARM) hosts, you must set PUPPETEER_EXECUTABLE_PATH (see "Environment details" below):

PUPPETEER_EXECUTABLE_PATH=$(echo ~/.cache/ms-playwright/chromium-*/chrome-linux/chrome) \
    ./tools/test-js-with-puppeteer _claude_<feature>_test

To run all existing Puppeteer tests, omit the test name argument.

Timeout: Tests can take 1–3 minutes each. Use a 300000ms timeout for the Bash tool.

3. Read the screenshots

Use the Read tool on each var/puppeteer/step-*.png file. Claude's multimodal vision will show the rendered page.

4. Analyze and report

Describe what you see — layout, colors, text content, positioning, any issues. Compare against what was expected.

Zulip displays any JS exceptions encountered as a pop-up, but you should also be able to get them from the puppeteer output.

5. Iterate if needed

If something is wrong, fix the source code (or adjust the test script), then re-run from step 2.

6. Clean up

Leave test files as untracked _claude_* files so you can reuse them when rebasing or iterating on the pull request. The _claude_ prefix is a convention to distinguish these from Zulip's committed test files. Do not commit them.

Environment details

• Architecture: On x86_64, Puppeteer's bundled Chrome works. On aarch64, it does NOT (fails with rosetta/ld-linux errors). Use uname -m to check. For aarch64, install Playwright's Chromium and point PUPPETEER_EXECUTABLE_PATH at it (shown in step 2).
• If the Playwright Chromium is missing, install it:

  npx --yes playwright install chromium
  

• Headless mode: Tests run headless (headless: true in common.ts). There is no display server.

Test infrastructure facts

• Server URL: http://zulip.zulipdev.com:9981/
• Login: common.log_in(page) uses credentials from var/puppeteer/test_credentials.json (auto-generated by the test harness). Default user is Desdemona (realm owner).
• Known users: common.fullname.cordelia, .othello, .hamlet
• Screenshots: saved to var/puppeteer/<name>.png
• Window size: 1400 x 1024
• Test data: The test database includes channels like "Verona", "Denmark", "Scotland" with topics and messages. Non-system user group hamletcharacters (members: Cordelia, Hamlet).
• zulip_test global: Only a limited set of internal functions are exposed — see web/src/zulip_test.ts. Functions like get_stream_id and get_user_id_from_name are available, but user_groups is not. Navigate to groups via URL hash routes or by clicking list items instead.
• Database reset: The test runner calls reset_zulip_test_database() and POST /flush_caches between test files, so each test file starts with a clean state.
• common.run_test() handles browser lifecycle, console log forwarding with source-map resolution, automatic failure screenshots, and logout at the end.

Available helpers (from web/e2e-tests/lib/common.ts)