Three Layer Testing

Three-Layer Testing Cycle

Every non-trivial code change ends with one pass of this cycle. No exceptions.

When to run

Run this skill when any of the following is true:

• The agent just finished writing or modifying code (new feature, bug fix, refactor, dep upgrade, config change that affects runtime).
• The user says 跑测试 / 回归 / 全量测试 / run tests / regression / test cycle / check if nothing is broken.
• Closing a task or PR-preparation moment.

Skip only if the change is pure docs / pure comments, or the user explicitly says "don't run tests".

Philosophy

Three concentric rings, fast → slow, cheap → expensive:

Unit catches most regressions for 1% of the time. Integration pins down the wiring. E2E certifies the user-visible experience — but it is expensive and flaky, so it only runs when the change is broad enough to warrant it.

Mandatory workflow

Copy this checklist and tick as you go. Step 4 is conditional — if not triggered, explicitly mark it skipped (out of trigger scope) in the report.

Testing cycle:
- [ ] Step 1  Discover framework + existing tests
- [ ] Step 2  Unit layer          (always; must pass)
- [ ] Step 3  Integration layer   (always; must pass)
- [ ] Step 4  System / E2E        (only when trigger matrix matches)
- [ ] Step 5  Triage + fix (test-first regression for every bug)
- [ ] Step 6  Report PASS/FAIL summary + bug table

Do not mark a step "done" without real tool output.

E2E trigger matrix

Before Step 4, evaluate the change against this matrix. Run E2E only if at least one row matches.

Skip E2E when the change is localized and none of the above triggers apply:

• Bug fix inside a single function with unit regression
• Adding a pure helper / utility
• Tightening a validator pattern
• Docstring / comment / rename inside one file
• Adding tests only

When skipping, the report's System row reads:

skipped (no trigger matched: change is localized to <module/file>)

If the decision is ambiguous, default to running E2E — the cost of a missed regression at that layer is higher than the cost of a 5-minute run.

Step 1 · Discover the project

Identify test framework + command. Use these heuristics, in order:

If nothing matches: ask the user for the test command once; don't guess.

For monorepos: run every sub-project's tests. Do not pick favourites.

Also inspect:

• tests/ layout — is there a unit/ integration/ system/ split? If yes, you can run each layer separately (e.g. pytest tests/unit then pytest tests/integration). If not, the layer distinction lives inside marker annotations (@pytest.mark.unit, etc.) or filename prefixes.
• CI config (.github/workflows/*.yml, .gitlab-ci.yml, Jenkinsfile) — the CI command is the canonical one; mirror it.

Step 2 · Unit layer

• Must run every cycle.
• If missing (no unit tests exist): flag it, write at least minimum unit tests for the code you just touched, then run.
• A passing unit suite that ignores the code you changed is not a pass — verify at least one assertion touches your change (coverage or grep).

Typical commands:

# pytest
pytest tests/unit -q --tb=line

# jest
npx jest --testPathPattern=unit --silent

# go
go test -run Unit ./... -count=1

Step 3 · Integration layer

• Uses an in-process client against the full app (TestClient / supertest / httptest / etc.) with an isolated temp DB — never the dev/production DB.
• All outbound HTTP to third-party services is mocked (respx, nock, wiremock, httpmock…).
• If integration tests don't exist yet: this is the highest-value layer to invest in; write at least one end-to-end happy-path plus the RBAC/error branches relevant to your change.

Typical commands:

pytest tests/integration -q --tb=line

npx jest --testPathPattern=integration --silent

Step 4 · System / E2E (conditional)

First check the E2E trigger matrix above. If no trigger matches, skip this step and record the skip reason in the report. Do not run E2E out of habit on localized changes.

When triggered, run only if all of the following are true:

1. A running service is reachable (check health endpoint / listening port).
2. A browser automation entry point is available — Cursor browser MCP (cursor-ide-browser), Playwright, Selenium, or a CLI equivalent.
3. Test fixtures/seed for known accounts exist (or the user authorises direct DB manipulation via a seed script).

If any runtime dep is missing, mark the system layer skipped in the report with a one-line reason. Do not block on it.

When running:

• Use a dedicated seed script to provision test accounts (seed command with idempotent reset); never log in with the user's personal credentials.
• Clean up afterwards (cleanup / fixture teardown) unless the account is intended to be long-lived.
• Focus journeys on the area the change affects (RBAC, auth flow, schema-bound page, etc.) — not necessarily every page. Full-site smoke is optional.
• Capture at least one screenshot per journey for the report.

Step 5 · Triage and fix

R1. Test-first regression (non-negotiable for any bug found)

For every failure / bug discovered:

1. Write a minimal test that reproduces the failure (a new test, or tighten an existing one). Run it — confirm RED.
2. Edit the production code to make that test GREEN.
3. Re-run the narrowest layer that contains the fix. If GREEN, run the wider suite.
4. Add a short comment on the test linking it to the bug ID (# Regression: BUG-P0-01 ...).

Never modify an existing passing test to accept the buggy behaviour. Never add an exception that silences a failure.

R2. Severity triage

| P0 | Functional or security breakage. Stop everything. Fix now + regression test + re-run full cycle. | | P1 | Consistency/UX/stability issue that has a workaround. Fix if < 15 min, else record in the bug table. | | P2 | Cosmetic, dead code, deprecation noise. Record only. |

R3. Classification matrix for failures

R4. Pass-rate ratchet

After your changes, total passed count must be >= the pre-change count. If it regresses (e.g. a previously passing test now fails and you can't figure out why), stop and investigate. Do not commit with lower pass count than before.

Step 6 · Report

Produce a concise report in the chat. Copy the template below:

## Test run summary (three-layer)

| Layer | Tests | Passed | Failed | XFail | Skipped | Time |
|-------|-------|--------|--------|-------|---------|------|
| Unit        | … | … | … | … | … | … s |
| Integration | … | … | … | … | … | … s |
| System      | … | … | … | … | … | … s *or* `skipped (out of trigger scope: <reason>)` |
| **Total**   | … | … | … | … | … | … s |

When System is skipped by the trigger matrix, also print one line explaining
which rule applied, e.g.:
`System E2E skipped — change is a 1-function fix in utils/sanitize.py, no trigger matched.`

### Bugs discovered & fixed this cycle

| ID | Severity | Symptom (1 line) | Root cause (1 line) | Fix file(s) | Regression test |
|----|----------|------------------|----------------------|-------------|-----------------|
| BUG-P0-01 | P0 | … | … | `foo/bar.py` | `tests/unit/test_bar.py::test_repro` |

### Still failing / known issues

- …  (or: none)

### Next suggested action

-   e.g. "run cycle on CI to confirm", or "no further action"

If everything PASSes and nothing was fixed, the table collapses to a single line — that is fine, still produce it.

Hard rules (apply always)

1. No fake PASS. Never claim 测试通过 / tests pass without the actual command output in the same turn. Cite exit code / summary line.
2. Mock all external I/O in unit + integration. If a test hits a real network endpoint, it belongs in E2E.
3. Never touch production / user data. Use a temp DB (tempfile, env DATABASE_URL=sqlite:///<tmp>) or a dedicated test schema.
4. No -x / fail-fast on the full suite. You want the full failure set for triage, not the first one.
5. Deterministic time — if a test depends on now(), mock the clock (freezegun, sinon.useFakeTimers, time.freeze).
6. Idempotent seed/cleanup — seed scripts should be safe to run 2× in a row without errors.
7. No emojis in code or test output files unless the user asked. Keep terminal output and commit messages ASCII-friendly.

Common anti-patterns to catch

• Tests that sleep() instead of waiting on a signal → replace with pytest-asyncio/waitUntil/signal.
• try: ... except Exception: pass inside a test — silences real failures.
• Shared mutable module state between tests — add an autouse reset fixture.
• Rate-limiter / timer fixtures that leak into the next test — reset in autouse fixture.
• E2E that logs in with the user's real credentials — forbidden; always use a dedicated test_* account managed by a seed script.
• Running E2E against production URLs — forbidden.

Degraded modes

Summary

Every code change → one full cycle → fix-with-regression-test → report.

The skill is done when:

• Unit PASS.
• Integration PASS.
• System PASS or explicitly skipped with a reason — out-of-trigger-scope for localized changes is a valid reason; missing runtime deps is also valid.
• Every discovered bug has both a code fix and a failing-then-passing regression test.
• A written report in the chat quoting real command output.