Sota Testing

SOTA Testing (2026)

Expert-level, language-agnostic rules for producing and auditing production test suites. Per-language runner/tooling details (pytest, go test, cargo test, vitest/jest) live in the language skills (sota-python, sota-golang, sota-rust, sota-javascript-typescript) — this skill defines the strategy, design discipline, and quality bar those tools execute against. Every rule states the why; every rules file ends with an audit checklist of yes/no questions and grep-able smells.

Purpose

Two consumers, one source of truth:

• BUILD mode — designing a test strategy or writing tests for new code: follow the rules as defaults, not suggestions. Deviate only with an explicit comment justifying the deviation.
• AUDIT mode — reviewing an existing suite: hunt violations using the audit checklists, classify by severity, report in the finding format below.

BUILD mode

1. Before writing tests, read the rules files relevant to the layer you are testing (see index). A service touching HTTP + DB + a message queue needs 01, 02, 03, 04.
2. Apply the top-10 non-negotiables (below) unconditionally.
3. Decide the suite shape FIRST (rules/01): what counts as a unit here, where the integration boundary is, which 3–10 flows deserve e2e. Write that decision down (CONTRIBUTING.md or a test README) so the next contributor doesn't relitigate it.
4. New test code is production code: same review bar, same lint rules, no TODO: assert something placeholders. A merged test with no assertion is worse than no test — it manufactures false confidence.
5. Write tests alongside the code, not after the PR is "done". For bug fixes, write the failing test first — it is the only proof the fix fixes anything.
6. Default to real dependencies in containers over mocks for anything with I/O semantics you don't own (DBs, brokers, caches) — see rules/04.
7. When generating code for a test that legitimately violates a rule (e.g. a sleep in a test that verifies a timeout), comment why inline.

AUDIT mode

Audit the suite, not just the tests: shape, doubles discipline, data management, CI health, and what is missing (untested risk) all count.

Severity conventions:

• Critical — the suite lies: assertion-free tests, tests that can't fail (always-green), mocks asserting mock behavior, disabled/skipped tests hiding known-broken production behavior, coverage gates gamed by meaningless tests.
• High — the suite is unreliable or unmaintainable at current trajectory: shared mutable state between tests, order-dependent tests, real time/network/randomness without injection, flaky tests un-quarantined and retried-to-green, e2e suite owning logic the unit layer should own, mocking internals so refactors break hundreds of tests.
• Medium — quality erosion: mystery-guest fixtures, multi-behavior tests, snapshot dumps nobody reviews, sleeps instead of waits, fixture data with irrelevant noise, missing negative-path tests on critical flows.
• Low — style/hygiene: weak names, redundant assertions, minor AAA violations, missing parameterization of near-duplicate tests.

Finding format (one per line):

file:line | rule-id | severity | finding and concrete fix

Example:

tests/orders_test.py:88 | 02-determinism | High | uses datetime.now(); inject a fixed clock so the test cannot fail at month boundaries
tests/api/user.spec.ts:12 | 03-mock-boundary | High | mocks internal UserValidator; test the real validator, mock only the HTTP gateway

End every audit with: findings table, top-3 risks, and a prioritized fix list (quick wins vs structural).

Rules index

Top-10 non-negotiables

1. Every test must be able to fail. A test that passes when the code under test is deleted or inverted is a Critical finding. Verify the failure mode when writing (break the code, watch it go red — TDD gives this for free).
2. Test behavior through public interfaces, not implementation. If a pure refactor (no behavior change) breaks the test, the test is wrong.
3. No real time, randomness, or network in unit tests. Inject clocks, seed or inject RNGs, fake the network. Nondeterminism is how flakes are born.
4. No shared mutable state between tests; no ordering dependence. Every test must pass alone, in any order, and in parallel with its siblings.
5. Mock only at architectural boundaries you own the interface to (your gateway/port), never internals, and don't mock types you don't own — wrap them, then fake the wrapper. Verify fakes against the real thing.
6. One logical behavior per test, named as a specification of that behavior (rejects_expired_card, not test_payment_2).
7. Integration tests use real dependencies (containerized DB/broker/cache), not in-memory lookalikes with different semantics. SQLite is not Postgres.
8. E2E is a small, curated, critical-path suite (smoke + money paths) with role/testid selectors and auto-waiting — never sleep(), never a dumping ground for cases a lower layer can cover.
9. Flaky tests are quarantined within a day, with an owner and an expiry — never silently retried-to-green forever, never deleted without a root-cause label (ordering / async / time / infra / test bug / real bug).
10. Coverage is a gap-finder, not a target. Ratchet it (never decrease), read the uncovered lines, and never write a test whose only purpose is to move the number.

Security-critical paths (authn/authz, crypto, input parsing, money/quota, tenancy, untrusted data) additionally require negative security tests at a higher coverage bar — see rules/09. A green functional suite proves nothing about IDOR, injection, or broken authz.