Fxa Test Independence

FXA Test Independence

Validate that tests pass both when run together and when run individually. A test that passes in the full suite but fails in isolation has a hidden dependency on execution order or shared state — that is a bug in the test, not the code.

Step 1: Identify the Target File

If $ARGUMENTS is provided, use it as the test file path.

Otherwise, check for recently changed test files:

git diff HEAD --name-only | grep -E '\.(spec|test)\.(ts|tsx)$'
git diff --cached --name-only | grep -E '\.(spec|test)\.(ts|tsx)$'

If multiple files are found, ask the engineer which to validate. If no file is found, ask for one explicitly.

Step 2: Extract Test Names

Read the target file and extract every it(...) / test(...) name. Build the full list before running anything.

Step 3: Present the Plan and Confirm

Show the engineer the list of tests to be validated and the commands that will be run. Do not proceed without confirmation.

File: packages/fxa-auth-server/lib/account.spec.ts
Tests found: 12

Will run:
  1. Full suite (all 12 tests together)
  2. Each test individually (12 isolated runs)
  Total runs: 13

Step 4: Run the Full Suite

Derive the package root from the test file path (e.g. packages/fxa-auth-server/lib/foo.spec.ts → packages/fxa-auth-server). Always cd into the package before invoking Jest so the local jest.config.* is picked up. Do NOT use nx test-unit — it runs the entire package suite regardless of the file argument.

Show the exact command, then run it:

cd <package-root> && npx jest <relative-path-to-spec> --no-coverage

Example: for packages/fxa-auth-server/lib/metricsCache.spec.ts:

cd packages/fxa-auth-server && npx jest lib/metricsCache.spec.ts --no-coverage

Record: pass/fail and any output for failing tests.

Step 5: Run Each Test in Isolation

For each test name extracted in Step 2, run it individually from the same package root:

cd <package-root> && npx jest <relative-path-to-spec> --testNamePattern="<exact test name>" --no-coverage

Record the result for each.

Step 6: Report Results

Output a results table:

Status codes:

• OK — passes in both contexts
• ORDER DEPENDENCY — passes in suite, fails in isolation; likely depends on state set by a prior test
• BROKEN — fails in both; implementation or mock issue
• FALSE POSITIVE — passes in isolation, fails in suite; likely pollutes shared state for other tests

For any non-OK result, include the failure output and a diagnosis:

Likely causes by status:

• ORDER DEPENDENCY: missing beforeEach reset, shared module-level variable mutated by a prior test, or singleton not re-initialised between tests
• FALSE POSITIVE: test mutates a shared mock or global without cleaning up in afterEach
• BROKEN: wrong mock return value, missing await, or the implementation under test has changed

Suggest a concrete fix for each failure. Do not fix automatically — present the diagnosis and let the engineer decide.