# Testing Strategy

> Use when planning or reviewing tests for a code change, choosing between unit, focused regression, integration, contract, end-to-end, performance, security, property-based, or agent-evaluation checks; classify change risk first and select the smallest evidence set that proves the behavior. Do not use for a single obvious test command, pure documentation changes, or a full security audit without a testing question.

- Skill: `anastasiyaw/testing-strategy` (Agent Skill)
- Install (CLI): `npx skillmds@latest add anastasiyaw/testing-strategy`
- Raw SKILL.md: https://api.skillmd.com/api/skills/anastasiyaw/testing-strategy/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: AnastasiyaW (https://skillmd.com/u/anastasiyaw)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/anastasiyaw/testing-strategy

---


# Testing Strategy

Testing is an evidence-selection problem, not a contest to run the largest
suite. Choose the smallest set that can falsify the changed behavior, then add
one higher-level check only when it covers a boundary the lower level cannot.
Keep execution environments reusable, but separate their evidence profiles:
`staging-smoke`, `security-proof`, `release-attestation`, and `nightly-stress`.
Use `harness-feedback` when a gate is reported as overloaded or misplaced.

## Workflow

1. Freeze the acceptance criteria as observable outcomes.
2. Inspect the changed files and classify the risk.
3. Select the lowest useful test level from the matrix below and name the profile.
4. Run the fast gate first. If it fails, fix the cause before adding more tests.
5. Add a focused regression test for a confirmed bug or a changed invariant.
6. Test real boundaries only when the change crosses them.
7. Keep security-proof and release-attestation checks out of staging-smoke unless
   the acceptance criteria explicitly require that evidence.
8. For high-risk or long-horizon work, use a fresh-context verifier and store
   the command, revision, result, and skipped checks in a durable artifact.
9. When a verified stage becomes the input to another stage, seal that boundary
   with commit/tree, contract, input/output digests, and a fresh verdict. Mark an
   unavailable external prerequisite as `BLOCKED`; do not rerun unrelated accepted
   code merely because the following environment is unavailable.

## Compact Matrix

| Change | Required evidence | Usually deferred |
|---|---|---|
| Docs, comments, formatting only | Link/lint check when relevant | Runtime suite |
| Pure function, local refactor | Fast checks + focused unit/regression tests | Full E2E, mutation |
| Parser, serializer, file, DB, API adapter | Fast + focused + one real boundary/integration check | Browser E2E unless user flow changes |
| Auth, permissions, migrations, concurrency, public API, deployment | Fast + focused + integration/contract + targeted smoke; independent review for non-trivial changes | Full load test unless performance is in scope |
| UI or user journey | Fast + component/focused checks + one stable E2E smoke | Large browser matrix |
| Release or performance claim | All applicable lower levels + fixed benchmark/security/release evidence | Nothing that is part of the claim |

The Stop hook runs the project's fast/default suite only when the working tree
contains code or test changes. Projects with a complex suite may declare
`.claude/test-policy.json`:

```json
{
  "fast": ["python", "-m", "pytest", "-q", "tests/unit"],
  "integration": ["python", "-m", "pytest", "-q", "tests/integration"],
  "release": ["python", "-m", "pytest", "-q"]
}
```

`fast` is the automatic Stop gate. `integration` is additionally selected for
high-risk changes when present. `release` is explicit or CI-only; do not make
every edit pay the release-suite cost. Optional `profiles` make the separation
explicit; a staging profile must not contain release-signing requirements.

## Test Kinds

- **Unit:** isolated behavior and invariants; fast and numerous.
- **Focused regression:** a minimal test that was red before a fix and green
  after it. Keep it when it protects a real contract.
- **Integration:** one real boundary such as a database, filesystem, queue, or
  external adapter. Use a local/test dependency, never production.
- **Contract:** provider/consumer schema and serialization expectations.
- **Smoke/E2E:** a small number of real user or release paths; keep them stable.
- **Property/fuzz:** invariants over generated inputs; use for parsers,
  normalizers, state machines, and edge-heavy algorithms.
- **Performance/security:** only when the change or release claim needs it;
  preserve a fixed workload and baseline.
- **Agent eval:** test task completion, tool selection, recovery, and safety on
  a versioned golden set. Deterministic assertions come first; an LLM judge is
  an additional signal, never the sole proof of code correctness.

## Agent Evidence Contract

An agent must report: revision, changed scope, commands actually run, exit
status, relevant counts, environment constraints, and checks not run with a
reason. “Tests passed” without command output or a durable evidence file is not
proof. A generated test is a candidate until it reproduces the failure or
asserts a stable contract; do not add broad snapshot tests merely to inflate
coverage.

For a confirmed bug use `bug-reproducer`: reproduce first, then fix, then run
the same test again. For a large/high-risk change use `proof-verify`: a fresh
context must produce the final verdict. For a safe structural refactor use
`refactoring-safely` and characterization tests before the transformation.

## Anti-Duplication Rules

- If a higher-level test finds a failure with no lower-level failure, add the
  smallest lower-level reproducer and keep the higher-level test only if it
  proves a distinct boundary.
- Do not run unit, integration, E2E, benchmark, and security suites by default
  just because they exist. Route by changed boundary and risk.
- Do not use retries, sleeps, snapshots, or `skip/xfail` to make red tests look
  green. A flaky test needs a cause, a bounded quarantine reason, or a fix.
- Do not claim release readiness from a fast suite alone.
- Do not treat a new external blocker as a failed upstream candidate. Verify the
  stage that changed; promote the same sealed input when the prerequisite arrives.

## Gotchas

- A green test suite proves only the exercised behavior; it does not prove
  absence of defects.
- Mocks can hide serialization and wiring failures. Keep one real boundary test
  for each important adapter.
- End-to-end tests are valuable but expensive and flaky; they should protect
  journeys, not duplicate every branch already tested below.
- Mutation testing is a periodic test-quality audit, not a per-edit gate. On
  Windows, verify the tool's runtime requirements before adding it to CI.
- Agent trajectories need task-outcome checks and tool-call checks, not only
  final-text similarity.
- A VM harness is an execution environment, not a release profile. Reuse the
  VM for staging and security checks, but attach signing and artifact identity
  checks only to `release-attestation`.

## Troubleshooting

| Symptom | Likely cause | Action |
|---|---|---|
| Stop gate runs in a docs-only change | Project has no Git-visible status or a broad override | Check `git status`; keep the default command scoped in `.claude/test-policy.json` |
| Fast suite passes, integration fails | A real boundary was changed or mocked away | Add/fix the boundary test; do not weaken the fast gate |
| E2E is flaky | Timing, shared state, browser/environment dependency | Make state isolated and waits explicit; reduce E2E to a stable smoke |
| Generated test passes without exposing the bug | Test asserts implementation details or never goes red | Reproduce the pre-fix failure and assert the user-visible invariant |
| Agent claims completion with skipped checks | Missing evidence contract or verifier | Record the skip reason and run `proof-verify` for high-risk work |
| Agent says the harness is too strict or blocks smoke | Profiles are coupled or a gate is misplaced | Invoke `harness-feedback`; capture the blocker, split profiles, and rerun the reduced smoke |
| A later audit asks to repeat a green earlier stage | Proof identity was not recorded, or its source/input changed | Check the stage receipt; reuse a sealed matching receipt or record a superseding stage |