snoodleboot-io Coverage Categories
A systematic approach to comprehensive test coverage. Work through these categories in order to ensure your tests catch all failure modes, edge cases, and security issues.
Category 1: HAPPY PATH
Definition: Expected inputs produce expected outputs with no errors.
Purpose: Verify core functionality works under normal conditions.
What to test:
- Valid, typical inputs
- Expected return values
- Expected side effects (DB writes, API calls)
- Expected state changes
Example:
def test_create_user_happy_path():
"""Test creating user with valid data succeeds."""
user_data = {
"email": "alice@example.com",
"name": "Alice",
"password": "SecurePass123!",
}
user = create_user(user_data)
# Assert expected return value
assert user.email == "alice@example.com"
assert user.name == "Alice"
assert user.id is not None
# Assert expected side effects
assert db.users.count() == 1
assert user.created_at is not None
Common mistakes:
- ❌ Only testing happy path (ignoring edge cases)
- ❌ Not verifying side effects (DB writes, logs, events)
- ❌ Using unrealistic inputs (test with production-like data)
Category 2: BOUNDARY VALUES
Definition: Test at the edges of valid ranges (min, max, exactly at limit, one over limit).
Purpose: Catch off-by-one errors, overflow issues, and boundary-related bugs.
What to test:
- Minimum valid value
- Maximum valid value
- Exactly at limit (e.g., string length exactly 255 if that's the limit)
- One below minimum (should fail)
- One above maximum (should fail)
- Empty collections (length 0)
- Single-item collections (length 1)
Example:
def test_username_length_boundaries():
"""Test username validation at boundary values."""
# Minimum length (3 chars) - valid
assert is_valid_username("abc") is True
# Below minimum (2 chars) - invalid
assert is_valid_username("ab") is False
# Maximum length (20 chars) - valid
assert is_valid_username("a" * 20) is True
# Above maximum (21 chars) - invalid
assert is_valid_username("a" * 21) is False
# Empty string - invalid
assert is_valid_username("") is False
def test_pagination_boundaries():
"""Test pagination at boundary values."""
# Page size = 1 (minimum)
results = get_users(page=1, size=1)
assert len(results) == 1
# Page size = 100 (maximum)
results = get_users(page=1, size=100)
assert len(results) <= 100
# Page size = 0 (invalid)
with pytest.raises(ValidationError):
get_users(page=1, size=0)
# Page size = 101 (over max)
with pytest.raises(ValidationError):
get_users(page=1, size=101)
Common boundary conditions:
- String length (min, max)
- Numeric ranges (min, max, 0, negative, positive)
- Collection size (0, 1, max)
- Date ranges (start, end, exactly at cutoff)
- Pagination (first page, last page, page beyond last)
Category 3: EMPTY / NULL / ZERO
Definition: Test with absent, null, zero, or empty values for each nullable input.
Purpose: Catch null pointer exceptions, division by zero, and missing data handling bugs.
What to test:
None/null/undefinedfor nullable fields- Empty strings (
"") - Empty collections (
[],{}) - Zero values (
0,0.0) - False boolean values
Example:
def test_calculate_average_with_empty_list():
"""Test average calculation with empty list."""
# Empty list should raise or return None (depending on design)
with pytest.raises(ValueError, match="Cannot calculate average of empty list"):
calculate_average([])
def test_get_user_by_email_with_none():
"""Test user lookup with None email."""
with pytest.raises(ValidationError, match="Email cannot be None"):
get_user_by_email(None)
def test_format_address_with_missing_fields():
"""Test address formatting with missing optional fields."""
address = {
"street": "123 Main St",
"city": "Springfield",
"state": None, # Optional field
"zip": "", # Empty string
}
# Should handle missing fields gracefully
formatted = format_address(address)
assert "123 Main St" in formatted
assert "Springfield" in formatted
# Should not include state or zip if missing
def test_divide_by_zero():
"""Test division with zero divisor."""
with pytest.raises(ZeroDivisionError):
divide(10, 0)
Checklist for nullable inputs:
- Test each nullable parameter with
None - Test each string parameter with
"" - Test each list parameter with
[] - Test each dict parameter with
{} - Test each numeric parameter with
0
Category 4: ERROR CASES
Definition: Test when dependencies fail, resources are unavailable, or external calls raise exceptions.
Purpose: Verify error handling, logging, and graceful degradation.
What to test:
- Database connection failures
- Network timeouts
- External API errors (4xx, 5xx)
- File not found
- Permission denied
- Resource exhaustion (out of memory, disk full)
- Invalid state transitions
Example:
def test_save_user_when_database_unavailable(mocker):
"""Test user save handles DB connection failure gracefully."""
# Mock database to raise connection error
mocker.patch.object(db, "save", side_effect=ConnectionError("DB unavailable"))
user = User(email="alice@example.com")
# Should raise specific error (not generic exception)
with pytest.raises(DatabaseError, match="Failed to save user"):
save_user(user)
# Should log error
assert "DB unavailable" in captured_logs
def test_fetch_user_when_api_returns_500(mocker):
"""Test user fetch handles API 500 error."""
mocker.patch("requests.get", return_value=MockResponse(status=500))
with pytest.raises(ExternalAPIError):
fetch_user_from_external_api("user123")
def test_process_payment_when_gateway_timeout(mocker):
"""Test payment processing handles gateway timeout."""
mocker.patch.object(
payment_gateway,
"charge",
side_effect=TimeoutError("Gateway timeout"),
)
result = process_payment(amount=100, card_token="tok_123")
# Should return failure result (not raise)
assert result.status == "failed"
assert "timeout" in result.error.lower()
# Should not charge user
assert payment_gateway.charge.call_count == 1 # No retries on timeout
Common error scenarios:
- Network failures (timeout, connection refused, DNS error)
- Authentication failures (invalid token, expired token)
- Authorization failures (insufficient permissions)
- Validation failures (invalid input format)
- Resource not found (404)
- Conflict (409 - duplicate, race condition)
- Rate limiting (429)
- Server errors (500, 502, 503)
Category 5: CONCURRENT / ORDERING
Definition: If function has state or depends on execution order, test with different orderings or concurrent access.
Purpose: Catch race conditions, deadlocks, and state corruption from concurrent access.
What to test:
- Concurrent reads
- Concurrent writes
- Read-modify-write races
- Order of operations (does A→B produce same result as B→A?)
- State transitions (valid vs invalid sequences)
Example:
import asyncio
import pytest
@pytest.mark.asyncio
async def test_concurrent_counter_increments():
"""Test counter handles concurrent increments correctly."""
counter = Counter()
# Run 100 concurrent increments
tasks = [counter.increment() for _ in range(100)]
await asyncio.gather(*tasks)
# Should be exactly 100 (no lost updates)
assert counter.value == 100
def test_order_state_transitions():
"""Test order state transitions enforce valid sequences."""
order = Order()
# Valid sequence: pending → confirmed → shipped → delivered
order.confirm()
assert order.status == OrderStatus.CONFIRMED
order.ship()
assert order.status == OrderStatus.SHIPPED
order.deliver()
assert order.status == OrderStatus.DELIVERED
def test_order_invalid_state_transition():
"""Test order prevents invalid state transitions."""
order = Order()
# Invalid: skip from pending directly to delivered
with pytest.raises(InvalidStateTransitionError):
order.deliver() # Must confirm and ship first
@pytest.mark.asyncio
async def test_concurrent_stock_updates():
"""Test concurrent stock updates don't oversell."""
product = Product(stock=10)
# 20 concurrent purchase attempts (should only succeed 10 times)
tasks = [product.purchase(quantity=1) for _ in range(20)]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Exactly 10 should succeed
successes = [r for r in results if not isinstance(r, Exception)]
assert len(successes) == 10
# Stock should be 0
assert product.stock == 0
When to test concurrency:
- ✅ Shared mutable state (counters, caches, inventories)
- ✅ Database writes (race conditions, lost updates)
- ✅ File writes (concurrent modifications)
- ✅ Resource pools (connection pools, thread pools)
When to skip:
- ❌ Pure functions (no side effects)
- ❌ Read-only operations
- ❌ Single-threaded contexts
Category 6: AUTHORIZATION BOUNDARIES
Definition: Test that functions enforce access control (who can call it, what they can access).
Purpose: Catch security vulnerabilities where unauthorized users can access restricted resources.
What to test:
- Unauthenticated access (no token)
- Authenticated but unauthorized (wrong role/permissions)
- Accessing resources owned by other users
- Privilege escalation attempts
Example:
def test_get_user_requires_authentication():
"""Test user retrieval requires authentication."""
# No auth token provided
with pytest.raises(Unauthenticated):
get_user(user_id="123", auth_token=None)
def test_update_user_requires_ownership():
"""Test user update requires ownership or admin role."""
alice = create_user(email="alice@example.com")
bob = create_user(email="bob@example.com")
# Bob tries to update Alice's profile (should fail)
with pytest.raises(Forbidden):
update_user(
user_id=alice.id,
data={"name": "Hacked"},
auth_user_id=bob.id,
)
def test_delete_order_requires_admin_role():
"""Test order deletion requires admin role."""
user = create_user(email="user@example.com", role="user")
admin = create_user(email="admin@example.com", role="admin")
order = create_order(user_id=user.id)
# Regular user cannot delete order
with pytest.raises(Forbidden):
delete_order(order_id=order.id, auth_user_id=user.id)
# Admin can delete order
delete_order(order_id=order.id, auth_user_id=admin.id)
assert get_order(order.id) is None
def test_list_users_filters_by_organization():
"""Test user list enforces organization boundary."""
org_a = create_organization(name="Org A")
org_b = create_organization(name="Org B")
alice = create_user(email="alice@a.com", org_id=org_a.id)
bob = create_user(email="bob@b.com", org_id=org_b.id)
# Alice should only see users in Org A
users = list_users(auth_user_id=alice.id)
assert len(users) == 1
assert users[0].id == alice.id
Authorization patterns to test:
- Resource ownership (user can only access their own data)
- Role-based access (admin vs user vs guest)
- Permission-based access (specific permissions required)
- Organization/tenant boundaries (multi-tenant isolation)
- Hierarchical access (manager can access subordinates' data)
Category 7: ADVERSARIAL INPUTS
Definition: Test with malicious or malformed inputs designed to exploit vulnerabilities.
Purpose: Catch security vulnerabilities (SQL injection, XSS, path traversal, etc.) and robustness issues.
What to test:
- SQL injection: SQL fragments in string inputs
- XSS: Script tags, JavaScript in text fields
- Path traversal:
../../../etc/passwdin file paths - Command injection: Shell metacharacters in system calls
- Unicode/emoji: Non-ASCII characters
- Control characters: Null bytes (
\0), newlines, tabs - Extremely long strings: 1MB strings, 10K-character names
- Format string attacks:
%s,%nin format strings - LDAP injection: LDAP query fragments
- XML injection: XML entities, CDATA
Example:
def test_search_users_with_sql_injection_attempt():
"""Test user search is protected against SQL injection."""
# Attempt SQL injection
malicious_query = "'; DROP TABLE users; --"
# Should NOT execute SQL, should treat as literal string
results = search_users(query=malicious_query)
# Should return empty results (no match) or error, NOT drop table
assert isinstance(results, list)
# Verify table still exists
assert db.table_exists("users")
def test_create_user_with_xss_in_name():
"""Test user creation sanitizes XSS in name field."""
malicious_name = "<script>alert('XSS')</script>"
user = create_user(email="test@example.com", name=malicious_name)
# Should escape or strip script tags
assert "<script>" not in user.name
assert user.name in [
"<script>alert('XSS')</script>", # Escaped
"alert('XSS')", # Stripped tags
"", # Rejected entirely
]
def test_read_file_with_path_traversal_attempt():
"""Test file read prevents path traversal."""
malicious_path = "../../../etc/passwd"
# Should reject or normalize path
with pytest.raises(SecurityError, match="Invalid file path"):
read_file(malicious_path)
def test_username_with_unicode_and_emoji():
"""Test username handles unicode and emoji correctly."""
unicode_names = [
"用户名", # Chinese characters
"Müller", # Umlaut
"José", # Accented character
"🎉🎊", # Emoji
"café", # Combining characters
]
for name in unicode_names:
# Should either accept or reject gracefully (not crash)
try:
user = create_user(email=f"{name}@example.com", name=name)
assert user.name is not None
except ValidationError as e:
# If rejected, should have clear error message
assert "character" in str(e).lower() or "unicode" in str(e).lower()
def test_comment_with_extremely_long_string():
"""Test comment field handles extremely long input."""
# 10MB string
huge_comment = "a" * (10 * 1024 * 1024)
# Should reject or truncate
with pytest.raises(ValidationError, match="too long|exceeds maximum"):
create_comment(text=huge_comment)
def test_email_with_null_bytes():
"""Test email validation rejects null bytes."""
malicious_email = "test@example.com\x00@evil.com"
# Should reject null bytes
with pytest.raises(ValidationError):
create_user(email=malicious_email)
Adversarial input checklist:
- SQL injection patterns (
'; DROP TABLE,1' OR '1'='1) - XSS patterns (
<script>,javascript:,onerror=) - Path traversal (
../,..\\, absolute paths) - Command injection (
;,|,&&, backticks) - Unicode edge cases (emoji, combining characters, RTL)
- Control characters (null byte, newline, carriage return)
- Extremely long strings (1MB+)
- Format string attacks (
%s,%n,%x)
Workflow
Step 1: Create Category Checklist
For the function/module being tested, create a checklist:
## Test Coverage for `create_order()`
- [ ] HAPPY PATH - Valid order creation
- [ ] BOUNDARY VALUES - Min/max item count, price ranges
- [ ] EMPTY/NULL - Empty items list, null user_id
- [ ] ERROR CASES - DB failure, payment gateway error
- [ ] CONCURRENT - Multiple orders from same user
- [ ] AUTHORIZATION - Only authenticated users can create orders
- [ ] ADVERSARIAL - XSS in product name, huge item counts
Step 2: Implement Tests for Each Category
Work through the checklist, implementing tests for each applicable category.
Step 3: Document Excluded Categories
If a category doesn't apply, document why:
## Test Coverage for `calculate_tax()`
- [x] HAPPY PATH - Valid tax calculation
- [x] BOUNDARY VALUES - 0 amount, max amount
- [x] EMPTY/NULL - null amount
- [x] ERROR CASES - Invalid tax rate
- [ ] CONCURRENT - N/A (pure function, no state)
- [ ] AUTHORIZATION - N/A (internal utility, no auth)
- [ ] ADVERSARIAL - N/A (validated numeric input only)
Step 4: Flag Coverage Gaps
If you can't test a category, explain why and how the risk is mitigated:
## Test Coverage for `process_payment()`
- [x] HAPPY PATH
- [x] ERROR CASES - Gateway timeout, card declined
- [ ] CONCURRENT - **GAP**: Concurrent payments from same user not tested
- **Mitigation**: Payment gateway handles idempotency
- **Risk**: Low (gateway prevents double-charge)
- **TODO**: Add test when we migrate to new gateway (Q3 2026)
Common Mistakes
❌ Mistake 1: Only Testing Happy Path
Problem: Tests pass with valid inputs but fail in production with edge cases.
Fix: Use this category framework to ensure comprehensive coverage.
❌ Mistake 2: Skipping Boundary Value Tests
Problem: Off-by-one errors, overflow bugs slip through.
Example:
- Password length validation allows 21-character password when max is 20
- Pagination returns 101 items when max is 100
Fix: Always test min, max, exactly-at-limit, and one-over-limit.
❌ Mistake 3: Not Testing Error Cases
Problem: Production errors crash the app instead of being handled gracefully.
Fix: Mock dependencies to throw exceptions and verify error handling.
❌ Mistake 4: Ignoring Security Tests
Problem: SQL injection, XSS, path traversal vulnerabilities discovered in production.
Fix: Test adversarial inputs for all user-facing functions.
Summary
Seven Categories for Comprehensive Coverage:
- HAPPY PATH - Normal operation
- BOUNDARY VALUES - Edge cases (min, max, limits)
- EMPTY/NULL/ZERO - Missing data
- ERROR CASES - Failures and exceptions
- CONCURRENT/ORDERING - Race conditions, state transitions
- AUTHORIZATION - Access control
- ADVERSARIAL - Security vulnerabilities
Usage:
- Create category checklist for each function/module
- Implement tests for applicable categories
- Document excluded categories with rationale
- Flag coverage gaps and mitigation strategies
Benefits:
- Systematic approach prevents missed edge cases
- Security testing is built-in
- Clear documentation of test coverage
- Easy to identify gaps and risks