Appsmithorg Appsmith Appsmith

Write & Verify Playwright Test

Prerequisites

Before starting, ensure Chromium is available:

cd app/client && npx playwright install chromium 2>/dev/null

Step 1 — Configure Environment

Run the configure-env.sh script to set up app/client/playwright/.env. It creates the file if missing and merges new values with any existing ones.

node .cursor/skills/write-and-verify-pw-test/scripts/configure-env.js \
  --PLAYWRIGHT_BASE_URL=https://target-dp.appsmith.com \
  --USERNAME=user@example.com \
  --PASSWORD=secret

Pass only the variables the caller provided. The script preserves existing values for anything not overridden. Supported variables:

PW_FLAG_OVERRIDES can go in .env via the script, or be passed inline when running tests (see Step 6).

Step 2 — Read Project Conventions

Before writing any spec, read these files for conventions (they are auto-applied when editing playwright/**/*.ts, but read them explicitly here):

• .cursor/rules/playwright.mdc — full conventions (POM rules, selectors, assertions, wait strategy)

Key rules to internalize:

1. Imports: import { test, expect } from "../../fixtures" (not @playwright/test)
2. Selectors: Use getByRole() > getByLabel() > getByTestId() > SELECTORS from constants. Never raw CSS.
3. No hard waits: No waitForTimeout, no networkidle. Wait for meaningful elements.
4. Auto-retrying assertions: await expect(locator).toBeVisible(), not expect(await locator.isVisible()).toBe(true)
5. Constants: API paths from playwright/constants/api-routes.ts, selectors from playwright/constants/selectors.ts, routes from playwright/constants/routes.ts. Never hardcode.
6. POMs: Constructor takes Page. No assertions in POMs. 1-5 lines per method.
7. API contract verification: Before any request.get/request.post, grep the server for the controller to confirm exact parameter names.

Step 3 — Determine Placement

Pick the tier

If the caller specifies a tier, use it. Otherwise, default to sanity.

Feature subdirectories are mandatory under sanity/ and regression/.

Decide: existing project, new project, or multiple specs

Read playwright.config.ts to understand the current projects and their setup chains.

Use an existing project when the new spec:

• Lives in a directory already covered by a project's testDir
• Doesn't need setup beyond what that project's dependency chain provides
• Example: a new widget test at tests/sanity/widgets/chart.spec.ts → use the sanity project

Create a new project when the new spec:

• Needs an expensive shared precondition not covered by existing setups (e.g., importing an app via API, connecting a specific datasource, seeding test data)
• Would slow down unrelated tests if its setup ran in a shared project
• Follow the setup/teardown/state-reader pattern from playwright.mdc (Parallelization section):
  1. Setup file in playwright/fixtures/<name>.setup.ts — performs the operation, writes state to playwright/.state/<name>.json
  2. Teardown file in playwright/fixtures/<name>.teardown.ts — cleans up
  3. State reader in playwright/helpers/<name>-state.ts — typed accessor for test files
  4. New project entry in playwright.config.ts with dependencies, testDir, and teardown

Split into multiple spec files when:

• The prompt describes multiple independent pages or concerns — one spec per page/concern
• Tests within a shared-setup project should be parallelizable — each file runs independently, reads shared state, navigates directly to its target
• A single spec would exceed ~10 tests — split by sub-feature
• Example: Git migration tests are split into migration-mysql.spec.ts, migration-postgres.spec.ts, migration-modal-form.spec.ts — all under regression/git/, all share the migration-setup project, each focused on one page

Rule of thumb: One test.describe per spec file, one concern per test(). If the prompt covers multiple concerns, prefer multiple small specs over one large one.

Step 4 — Write the Spec

Write the spec file following these patterns. Reference existing examples:

Simple spec (smoke-level):

import { test, expect } from "../../fixtures";
import { ROUTES } from "../../constants/routes";

test.describe("Smoke — Feature Name", () => {
  test("behavior description in lowercase", async ({ page }) => {
    await page.goto(ROUTES.applications);
    await expect(page.getByRole("button", { name: /new/i })).toBeVisible();
  });
});

Spec with fixture workspace/app (sanity/regression):

import { test, expect } from "../../fixtures";
import { SELECTORS } from "../../constants/selectors";

test.describe("Feature — Specific Area", () => {
  test("filters table by country", async ({ page, app }) => {
    await page.goto(app.url);
    await expect(page.locator(SELECTORS.widgetInDeployed("textwidget")).first()).toBeVisible();
    // test logic here
  });
});

If a POM is needed, create it under playwright/page-objects/ (or components/ for reusable widgets). Follow existing patterns like home.page.ts.

Step 5 — Lint Check

After writing each file, run:

cd app/client && npx eslint <spec-file-path> --ext .ts

Fix any issues before proceeding.

Step 6 — Run the Test

Auth chain (automatic — do not skip)

The config chains setup projects: signup-setup → setup → your test project. This handles fresh deployments automatically:

• signup-setup: Creates the user if needed (handles empty instance onboarding, legacy signup, and "already exists" gracefully)
• setup: Logs in and saves auth state to playwright/auth/user.json
• test project: Runs your specs with the saved auth state

This chain runs automatically when you use --project. Never bypass it by running test files directly without a project flag.

Determine the correct project

Critical: Each project in playwright.config.ts has its own testDir, dependencies, and sometimes testIgnore. The spec must be run under the project whose testDir matches and whose dependency chain includes the required setup.

Read playwright.config.ts and match your spec's directory to a project. Current mapping:

Why this matters: Some projects have extra setup phases (e.g., regression-git depends on migration-setup which imports a Git app and creates datasources). Specs in those directories read shared state produced by the setup (e.g., loadMigrationState()). Running them under the wrong project skips that setup and they fail immediately.

When writing a new spec that needs a custom setup project (e.g., it relies on pre-imported data, a connected datasource, or shared app state):

1. Check if an existing setup project covers the need
2. If not, create a new setup/teardown pair following the pattern in playwright.mdc (Parallelization section)
3. Add the new project to playwright.config.ts with proper dependencies and testDir
4. Update this mapping table

If the config has changed since this skill was written, always read playwright.config.ts to get the current project list. Don't rely on this table alone.

Run command

cd app/client && npx playwright test <spec-file-path> --project=<project>

If PW_FLAG_OVERRIDES was provided:

cd app/client && PW_FLAG_OVERRIDES='{"flag_name": true}' npx playwright test <spec-file-path> --project=<project>

Timeout: Set block_until_ms to at least 120000 (2 min). Tests can take 60s+ with signup + auth setup on first run against a fresh deployment.

Step 7 — Retry Loop (max 3 attempts)

If the test fails, classify the failure:

Test bug signals (fix the spec):

• locator.click: Target closed → wrong selector or premature navigation
• Timeout + element exists in DOM but not visible → missing wait or wrong locator
• strict mode violation → locator matches multiple elements, needs .first() or refinement
• Error: expect(received).toHaveText(expected) where expected value is clearly wrong → bad test data
• Import errors, TypeScript errors → code issue in the spec
• waiting for locator → selector doesn't match anything, check the actual page

Action: Read the error output, fix the spec, re-run. This is attempt N+1.

Product bug signals (stop and diagnose):

• expect(received).toHaveText(expected) where expected is correct but received shows clearly wrong app behavior
• net::ERR_CONNECTION_REFUSED → deployment is down
• 401 / 403 after successful auth setup → permission regression
• Visual mismatch where the test selector is correct but the UI renders wrong content
• Server returns 500 on API calls

Action: Stop retrying. Switch to the diagnose-pw-failure skill. See .cursor/skills/diagnose-pw-failure/SKILL.md.

Ambiguous failures:

Default to "fix the spec" for the first 2 attempts. If the same assertion fails 3 times with the same received value, it's likely a product bug — switch to diagnosis.

Step 8 — Summary

After the test passes (or after exhausting retries), provide a summary:

On success:

## Playwright Test Result: PASS

**Spec**: playwright/tests/sanity/widgets/table-filter.spec.ts
**Deployment**: https://my-dp.appsmith.com
**Project**: sanity
**Attempts**: 2 (1 fix applied: wrong selector for filter button)

### What was tested
- Table widget renders with data
- Filtering by country "Ba" shows Bangladesh

On failure (product bug):

## Playwright Test Result: FAIL (product bug suspected)

**Spec**: playwright/tests/sanity/widgets/table-filter.spec.ts
**Deployment**: https://my-dp.appsmith.com
**Attempts**: 3 (spec verified correct)

### Failure
- Table filter returns empty results when filtering by "Ba"
- Expected: rows containing "Bangladesh"
- Received: 0 rows
- Screenshot: playwright/results/<test-name>/screenshot.png

### Diagnosis
[See diagnose-pw-failure output]

Source: appsmithorg/appsmith — distributed by TomeVault.