# Agni

> Testing standards — unit/integration/e2e strategy, pytest and vitest conventions, and LLM eval suites. Use when writing tests, discussing test coverage or TDD, fixing flaky tests, or building evals for prompts and model outputs.

- Skill: `arjuncrevathi/agni` (Agent Skill)
- Install (CLI): `npx skillmds@latest add arjuncrevathi/agni`
- Raw SKILL.md: https://api.skillmd.com/api/skills/arjuncrevathi/agni/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: arjuncrevathi (https://skillmd.com/u/arjuncrevathi)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/arjuncrevathi/agni

---


# Agni — Trial by Fire (Testing)

Agni governs the fire every change must pass through before it is trusted.

## The pyramid

- Many unit tests (fast, no I/O), some integration tests (real DB via testcontainers/Docker), few e2e tests (critical user flows only).
- Unit suite runs in seconds and on every save; the full suite runs in CI on every PR (see `brahma`).
- Coverage is a smoke alarm, not a goal: alert on drops, don't chase 100%. Untested error paths matter more than tested getters.

## What to test

- Test behavior through public interfaces, not implementation. A refactor that preserves behavior should break zero tests.
- Every bug fix gets a regression test that fails on the old code, in the same PR (see `durga`).
- Name tests as claims: `test_expired_token_returns_401`, not `test_auth_2`.
- Cover the unhappy paths: empty input, timeout, malformed data, permission denied.

## Tools

- Python: `pytest`. Shared setup lives in fixtures (`conftest.py`), not copy-paste. Use `pytest.mark.parametrize` for input matrices; `factory-boy`/builders for test data.
- JS/TS: `vitest` for unit/component (with Testing Library), `playwright` for e2e. Query by role/label, not CSS selectors.
- Mock external services at the boundary only — the HTTP layer (`respx`, `msw`) or your adapter interface. Never mock your own domain logic; if you must, the design is wrong (see `vishwakarma`).

## Flakes

- No flaky tests. A test that fails 1-in-20 runs gets fixed today or deleted — retrying-until-green trains everyone to ignore red.
- Usual demons: real time (`sleep`, `datetime.now()` — inject clocks), shared state between tests, unawaited async work, order dependence. Run with `pytest -p no:randomly` off / `--random-order` to expose ordering bugs.

## AI-native specifics

- Every prompt gets an eval suite: a golden dataset (20+ real, versioned examples) with expected outputs or grading criteria, runnable via one command.
- Assert on structure, not exact wording: parse the JSON, check required fields, check facts/citations present — never `assert output == "exact string"` against a model.
- LLM-as-judge with caution: pin the judge model and version, write a rubric, spot-check judge verdicts against human labels regularly. Prefer deterministic checks (schema, regex, contains) wherever possible.
- Run evals in CI on any PR touching prompts, model config, or the prompt layer. Set a pass threshold; a score drop blocks merge.
- Track eval scores over time (per prompt, per model version) in the prompt changelog (see `saraswati`) so regressions are visible, not vibes.
- Keep eval runs cheap: temperature 0, small model for smoke evals, full suite nightly.

## Before merging — checklist

- [ ] New behavior covered by tests at the right pyramid level
- [ ] Unhappy paths tested, not just the golden path
- [ ] No mocks of internal domain logic
- [ ] Suite passes 3 consecutive runs locally (no flakes)
- [ ] Prompt/model changes ran the eval suite; score recorded