Debug Test

Debug Test

Debug and fix failing Playwright tests using MCP as the primary diagnostic tool. Works for a single spec, a directory, or a tag. Merged from openshift-ui-tests-template/debug-test.md and test-fix-cycle.md.

Before Starting

1. Check that frontend/e2e/.env exists. If missing, copy frontend/e2e/.env.example to frontend/e2e/.env and tell the user to fill in their cluster values before continuing.
2. Read .claude/e2e-context.md for project conventions, patterns, and rules — page objects, selectors, fixtures, cleanup, waits, and things to never do. That file is the single source of truth for how Playwright tests should be structured.

Input

• Spec file: /debug-test e2e/tests/console/cluster-settings/upstream-modal.spec.ts
• Test name: /debug-test "Verify console login"
• Directory: /debug-test e2e/tests/helm/
• Project: /debug-test --project=helm
• Optional workers: append --workers=N (default: 4)

Examples:

/debug-test e2e/tests/console/cluster-settings/upstream-modal.spec.ts
/debug-test e2e/tests/helm/ --workers=2
/debug-test --project=topology

Workflow

Phase 1: Run

1. Build command from input:
   • File: npx playwright test <file> --retries=0 --workers=1 --reporter=list
   • Directory: npx playwright test <directory> --retries=0 --workers=<N> --reporter=list
   • Project: npx playwright test --project=<name> --retries=0 --workers=<N> --reporter=list
   • For developer tests: npx playwright test --project=<name>-developer --retries=0 --workers=1 --reporter=list
2. Run and capture output
3. If all pass: report success. If single file, run 2 more times to confirm stability. Done.
4. Parse failures: test name, file path, error message

Phase 2: Diagnose

For each failure, use MCP to identify root cause:

1. Resize viewport to 1920×1080
2. Navigate to the failing page
3. Snapshot accessibility tree — compare with test selectors
4. Interact — reproduce the failing action
5. Check console errors and network failures
6. Classify: Selector Changed, Timing Issue, API/Auth Failure, Functional Regression, Test Code Bug

Phase 3: Fix

For each fixable failure, in priority order:

1. Import/type errors first — may cause cascading failures
2. Stale selectors — use MCP snapshot to find correct selector
3. Timing issues — add robustClick() or waitForLoadingComplete()
4. Missing awaits — add await
5. Functional regressions — test.skip(true, 'reason') after confirming with MCP

Fix in the correct layer: selectors and waits in page objects, assertions in test files. Never put locators directly in test files when a page object exists.

Run individual test after each fix to verify. If unfixable after 2 attempts: test.skip(true, 'Descriptive reason').

Phase 4: Validate and Report

1. Re-run the full target (file, directory, or tag). If new failures, repeat phases 2-3.
2. For single files: run 3 consecutive times to confirm stability.
3. Output summary:

Debug Summary: <target>
  Total tests: N
  Passing: X
  Fixed: Y
  Skipped: Z

  Fixes applied:
    - <file>: <fix description>

  Tests skipped:
    - <file>: <reason>

Phase 5: What We Learned

After the fix is validated (Phase 4 passes), evaluate whether the root cause reveals a gap in the shared knowledge files. Most failures are one-off test bugs (wrong selector, missing await) and don't need documentation changes. Only flag patterns that would prevent future tests from hitting the same class of problem.

1. Classify the root cause. Ask: "Would reading e2e-context.md, migration-context.md, or the skill instructions have prevented this failure?" If yes, there's a gap.

2. Check for pattern, not incident. A gap is worth documenting when:

   • The fix relies on behavior not mentioned anywhere (e.g., PF component wrapping, API propagation delays, fixture lifecycle quirks)
   • The existing guidance actively led the agent astray (e.g., "use getByTestId" without warning about wrapper divs)
   • The same class of failure is likely to recur in other tests

   Skip if: the fix was a typo, a missing import, a stale selector, or a test-specific logic error.

3. Propose the change. If a gap exists, present:

   Documentation improvement:
     File: .claude/e2e-context.md (or migration-context.md, or skill SKILL.md)
     Section: <existing section name>
     What to add: <1-3 sentence description of the pattern>
     Why: <what went wrong and how this prevents it>
   

4. Ask the user. Do not apply changes without approval. Present the proposal and wait for confirmation.

5. Apply if approved. Spawn a subagent (haiku) with specific edit instructions: the target file, the section, and the exact content to add. The subagent must only edit the approved file and section. After the subagent returns, verify the change matches the proposal. If it edited files outside the approved target, revert.

Skip this phase entirely if no tests were fixed (all passed on first run or all were skipped).

Troubleshooting

Playwright MCP not connected

If MCP tools fail with "tool not found" or "connection refused": diagnose from error messages alone. Warn: "MCP not available — selector fixes may be inaccurate without live verification." Focus on fixes that don't require live inspection (missing awaits, type errors, obvious selector typos).

No cluster reachable

If npx playwright test fails with login/connection errors on every test: this is an infrastructure issue, not a test bug. Report it to the user and suggest checking cluster access, BRIDGE_BASE_ADDRESS, and storageState files.

Playwright not installed

If npx playwright test fails with "Cannot find module": the Playwright foundation hasn't been set up yet. Tell the user to complete the infrastructure setup first.

Flaky test (passes sometimes, fails sometimes)

If a test passes on re-run without any fix: it's flaky. Use MCP to identify the timing-sensitive interaction, then fix with robustClick(), waitForLoadingComplete(), or a more specific waitFor() condition. Never mask flakiness with retries.

Rules

• MCP first — always try browser tools before guessing at selectors
• Fix in the right layer
• Always use --retries=0 during the cycle
• Run cd frontend && yarn eslint <fixed-files> after fixes
• DO NOT commit — the user handles git operations