Agni — Trial by Fire (Testing)
Agni governs the fire every change must pass through before it is trusted.
The pyramid
- Many unit tests (fast, no I/O), some integration tests (real DB via testcontainers/Docker), few e2e tests (critical user flows only).
- Unit suite runs in seconds and on every save; the full suite runs in CI on every PR (see
brahma). - Coverage is a smoke alarm, not a goal: alert on drops, don't chase 100%. Untested error paths matter more than tested getters.
What to test
- Test behavior through public interfaces, not implementation. A refactor that preserves behavior should break zero tests.
- Every bug fix gets a regression test that fails on the old code, in the same PR (see
durga). - Name tests as claims:
test_expired_token_returns_401, nottest_auth_2. - Cover the unhappy paths: empty input, timeout, malformed data, permission denied.
Tools
- Python:
pytest. Shared setup lives in fixtures (conftest.py), not copy-paste. Usepytest.mark.parametrizefor input matrices;factory-boy/builders for test data. - JS/TS:
vitestfor unit/component (with Testing Library),playwrightfor e2e. Query by role/label, not CSS selectors. - Mock external services at the boundary only — the HTTP layer (
respx,msw) or your adapter interface. Never mock your own domain logic; if you must, the design is wrong (seevishwakarma).
Flakes
- No flaky tests. A test that fails 1-in-20 runs gets fixed today or deleted — retrying-until-green trains everyone to ignore red.
- Usual demons: real time (
sleep,datetime.now()— inject clocks), shared state between tests, unawaited async work, order dependence. Run withpytest -p no:randomlyoff /--random-orderto expose ordering bugs.
AI-native specifics
- Every prompt gets an eval suite: a golden dataset (20+ real, versioned examples) with expected outputs or grading criteria, runnable via one command.
- Assert on structure, not exact wording: parse the JSON, check required fields, check facts/citations present — never
assert output == "exact string"against a model. - LLM-as-judge with caution: pin the judge model and version, write a rubric, spot-check judge verdicts against human labels regularly. Prefer deterministic checks (schema, regex, contains) wherever possible.
- Run evals in CI on any PR touching prompts, model config, or the prompt layer. Set a pass threshold; a score drop blocks merge.
- Track eval scores over time (per prompt, per model version) in the prompt changelog (see
saraswati) so regressions are visible, not vibes. - Keep eval runs cheap: temperature 0, small model for smoke evals, full suite nightly.
Before merging — checklist
- New behavior covered by tests at the right pyramid level
- Unhappy paths tested, not just the golden path
- No mocks of internal domain logic
- Suite passes 3 consecutive runs locally (no flakes)
- Prompt/model changes ran the eval suite; score recorded