Testing Strategies

Kailash Testing Strategies

3-tier testing strategy with real infrastructure policy for Kailash applications.

Sub-File Index

• test-3tier-strategy - Complete 3-tier guide: tier definitions, fixture patterns, CI/CD integration

3-Tier Strategy

Real Infrastructure Policy (Tiers 2-3)

Why: Mocking hides database constraints, API timeouts, race conditions, connection pool exhaustion, schema migration issues, and LLM token limits.

What to use instead: Test databases (Docker containers), test API endpoints, test LLM accounts (with caching), temp directories.

Key Fixtures

@pytest.fixture
def db():
    """Real database for testing."""
    db = DataFlow("postgresql://test:test@localhost:5433/test_db")
    db.create_tables()
    yield db
    db.drop_tables()

@pytest.fixture
def runtime():
    return LocalRuntime()

Test Organization

tests/
  unit/          # Mocking allowed
  integration/   # Real infrastructure
  e2e/           # Full system
  conftest.py          # Shared fixtures

Component Testing Summary

Regression Test Design

Regression tests lock in bug fixes. They MUST exercise the actual code path -- call the function, assert the raise or return value. Source-grep tests are BLOCKED as the sole assertion because they pin the implementation, not the contract: when the fix moves to a shared helper (the right refactor), the grep breaks even though the protection is still in place.

# Behavioral (survives refactors)
@pytest.mark.regression
def test_null_byte_rejected():
    parsed = urlparse("mysql://user:%00x@h/db")
    with pytest.raises(ValueError, match="null byte"):
        decode_userinfo_or_raise(parsed)

# Source-grep (BLOCKED as sole assertion)
def test_null_byte_exists_in_source():
    assert "\\x00" in open("src/myapp/db/connection.py").read()

See rules/testing.md "MUST: Behavioral Regression Tests Over Source-Grep" for the full rule and rationale.

Optional Dependency Testing

Tests that exercise optional extras (e.g., [hpo], [redis], [vault]) MUST guard against the dependency being absent. Use pytest.importorskip at module or class scope so the test is skipped (not failed) in CI environments that don't install the extra.

# At module level — skips entire file if optuna is missing
optuna = pytest.importorskip("optuna", reason="optuna required for HPO tests")

class TestSuccessiveHalving:
    @pytest.mark.asyncio
    async def test_pruning(self):
        # optuna is guaranteed available here
        ...

Why: Base CI installs core dependencies only. A test that imports an optional extra without a skip guard fails every CI matrix entry, blocking unrelated PRs. pytest.importorskip is the standard mechanism — it imports the module if available and calls pytest.skip if not.

Where to place the guard: Before the first use of the optional module — typically at module scope (before the test class) or inside a fixture. Placing it inside a test function body is too late if the class-level setup already depends on the import.

Critical Rules

• Tier 1: Mock external dependencies
• Tier 2-3: Real infrastructure, no @patch/MagicMock/unittest.mock
• Docker for test databases
• Clean up resources after every test
• Cache LLM responses for cost control
• Run Tier 1 in CI always; Tier 2-3 optionally
• Never commit test credentials

Running Tests

pytest tests/unit/        # Fast CI
pytest tests/integration/ # With real infra
pytest tests/e2e/         # Full system
pytest --cov=app --cov-report=html  # Coverage

Related Skills

• 07-development-guides - Testing patterns
• 17-gold-standards - Testing best practices
• 02-dataflow - DataFlow testing
• 03-nexus - API testing

Support

• testing-specialist - Testing strategies and patterns
• tdd-implementer - Test-driven development
• dataflow-specialist - DataFlow testing patterns

Converted and distributed by TomeVault — claim your Tome and manage your conversions.