Test Or Bug

Test or Bug

Diagnose failing E2E tests to determine whether the failure indicates a code bug or a test issue.

When to Use

Invoke when user says:

• "test or bug"
• "is this a test issue"
• "why is this test failing"
• "diagnose test failure"
• "debug e2e"
• "analyze failing test"
• "is this a bug or flaky test"

Input: Path to a failing E2E test spec file (e.g., playwright/tests/session-creation.spec.ts)

Workflow Overview

Expected total duration: 2-4 minutes depending on test complexity.

For detailed phase instructions: references/diagnostic-phases.md

Diagnostic Decision Tree

Use this flowchart to classify failures:

E2E Test Failing?
│
├─ Test PASSES when skill runs it?
│  └─ FLAKY TEST - Likely test issue (race condition, timing)
│     → Check for missing waits, hardcoded timeouts
│
├─ In CI only, passes locally?
│  ├─ Worker/async related? → Pattern 4 (emit_automation_event)
│  ├─ Route cleanup issue? → Pattern 5 (fixture import)
│  └─ Parallel collision? → Pattern 12 (worker-isolated dates)
│
├─ Timeout error?
│  ├─ Selector not found? → Check if element exists in code
│  ├─ Network timeout? → Check API endpoint exists
│  └─ Generic timeout? → Pattern 7 (use specific waits)
│
├─ Assertion failure?
│  ├─ Expected value changed? → Likely CODE BUG
│  ├─ Wrong expected value in test? → TEST ISSUE
│  └─ Data-dependent? → Check seeding
│
├─ Database constraint error?
│  └─ Pattern 11 (missing required fields, unique violations)
│
├─ Auth timeout?
│  └─ Pattern 1 (chromium vs chromium-clean project)
│
└─ Test order matters?
   └─ Pattern 2 (test isolation, shared state)

Pattern references from: claude-patterns/playwright-best-practices.md

Verdict Classification

CODE BUG Indicators

TEST ISSUE Indicators

BOTH (Compound Verdict)

Sometimes a failing test reveals BOTH issues:

• Test has pattern violations AND code has a bug
• Report both with separate severity levels
• Recommend fixing test first if it masks the real failure

UNCERTAIN

If signals are ambiguous:

• Report confidence level (Low/Medium/High)
• Present evidence for each hypothesis
• Recommend manual investigation

Phase Execution

Phase 1: Run Test

npx playwright test <test-file> --reporter=list --timeout=60000 --retries=0

Flags explained:

• --reporter=list: Clear output format
• --timeout=60000: 60s timeout for faster diagnosis
• --retries=0: No retries to see actual failure

If test PASSES:

1. Note this as potential flaky test
2. Run again 2 more times to confirm
3. If inconsistent (some pass, some fail), classify as FLAKY

Phase 2: Error Analysis

Read the playwright output and classify:

1. Timeout: TimeoutError, waiting for selector, waiting for
2. Assertion: expect(received).toBe(expected), toHaveText, toBeVisible
3. Network: net::ERR_, ECONNREFUSED, fetch failed
4. Database: duplicate key, violates foreign key, null value
5. Auth: unauthorized, 401, login failed

Phase 3: Code Investigation

# Check recent changes to relevant files
git log --oneline -10 -- <relevant-paths>

# Diff against last known good state
git diff HEAD~5 -- <relevant-paths>

Identify:

• Files touched by the test (components, services, hooks)
• Recent commits that modified those files
• Changes that could cause the failure

Phase 4: Pattern Check

Read claude-patterns/playwright-best-practices.md and check test against:

• Pattern 3: DB seeding vs UI creation
• Pattern 5: Route handler cleanup
• Pattern 7: Specific waits vs networkidle
• Pattern 8: Silent pass anti-patterns
• Pattern 12: Worker-isolated dates

Phase 5: Verdict & Fix

Present findings using format from references/verdict-template.md

Quick Reference

Commands:

# Run specific test
npx playwright test playwright/tests/<name>.spec.ts

# Run with trace
npx playwright test <test> --trace on

# Show last report
npx playwright show-report

Key Pattern Files:

• claude-patterns/playwright-best-practices.md - Primary diagnostic source
• playwright/fixtures/seedHelpers.account.ts - Seeding helpers
• playwright/fixtures/testIsolation.ts - Worker-isolated dates
• playwright/utils/timeouts.ts - TIMEOUTS constants

Common Fixes:

References

• references/diagnostic-phases.md - Detailed phase execution
• references/common-failures.md - Failure patterns and fixes
• references/verdict-template.md - Output format

Version History

• v1.1.0 (2025-01-18): AI optimization updates

  • Add blockquote summary after title

• v1.0.0 (2025-01-15): Initial release

  • 5-phase diagnostic workflow
  • Decision tree from playwright-best-practices.md
  • Support for CODE BUG, TEST ISSUE, BOTH, and UNCERTAIN verdicts
  • Flaky test detection