Repo Pipeline Setup
Three-phase repo onboarding: Phase A bootstraps test infrastructure (language detection, framework detection, test runner + coverage, smoke tests, gold-standard templates). Phase B wires the CI/CD pipeline (Codecov, SHA pinning, SBOM, vulnerability scanning, security backstop, Dependabot, commit signing, OpenSSF Scorecard, CodeScene, GitGuardian). Phase C generates a project CLAUDE.md by analyzing the repo's tech stack, structure, commands, conventions, and CI configuration — so every new Claude Code session starts with full project context.
When to Use
Phase A (Test Infrastructure):
- Onboarding an existing repo that has application code but no test suite
- Starting a new project and want test infrastructure from the start
- CI audit found missing test infrastructure (e.g., Codecov marked N/A)
Phase B (CI/CD Pipeline):
- Onboarding a new repo into the CI pipeline
- Adding or fixing Codecov, pinact, or GitGuardian for existing repos
- Deploying pipeline changes across multiple repos at once
- Fixing cross-platform CI failures (lightningcss, npm ci, vitest coverage, Swift iOS-only)
- Auditing CI pipeline completeness across the portfolio
- Adding SBOM generation or build provenance attestations
- Setting up SSH commit signing or troubleshooting signature issues
- Configuring Dependabot security alerts or version updates
- Deploying OpenSSF Scorecard or SECURITY.md
- Setting up CodeScene behavioral code analysis on PRs
- Adding security scanning CI backstop (Semgrep, Checkov, Zizmor) for defense-in-depth
Phase C (CLAUDE.md):
- Onboarding a new repo — auto-runs after Phase B completes
- Repo has no
.claude/CLAUDE.mdor it contains only boilerplate - User asks to generate or refresh a project's CLAUDE.md
- Significant infrastructure changes (new test framework, CI additions) made CLAUDE.md stale
Phase A: Test Infrastructure
Bootstrap test infrastructure for repos with testable code but no tests. Detects language and framework, installs the test runner + coverage provider, creates config, and generates a smoke test + gold-standard template test.
When to Use
- Onboarding an existing repo that has application code but no test suite
- Starting a new project and want test infrastructure from the start
- CI audit found missing test infrastructure (e.g., Codecov marked N/A)
A0. Precondition Check
Before running detection, verify the repo has testable application code.
A repo is "testable" when BOTH conditions are met:
- At least one language config file exists:
package.json,go.mod,Cargo.toml,pyproject.toml,setup.py,requirements.txt,Package.swift, or a*.xcodeprojdirectory - At least one non-test source file exists in that language (
.ts,.tsx,.js,.jsx,.py,.go,.rs,.swift)
Exclude from source file count: node_modules/, vendor/, .git/, dist/, build/, generated files.
If NEITHER condition is met (no config file AND no source files), stop and report:
"This repo has no testable application code. Test infrastructure is not applicable. Consider shellcheck for shell scripts or JSON schema validation for config files."
A1. Detection
A1a. Language Detection
Detect languages in order of confidence. Check config files first (highest signal), then fall back to file extension counts.
Primary signal — config files:
| Config File | Language |
|---|---|
package.json |
TypeScript/JavaScript |
go.mod |
Go |
Cargo.toml |
Rust |
pyproject.toml, setup.py, requirements.txt |
Python |
Package.swift, *.xcodeproj (directory, not file) |
Swift |
Fallback — file extension count (when no config file found for a language):
| Extensions | Language |
|---|---|
.ts, .tsx, .js, .jsx |
TypeScript/JavaScript |
.go |
Go |
.rs |
Rust |
.py |
Python |
.swift |
Swift |
Mixed repos: Detect ALL languages present. Scope each language's setup to its root directory:
- Find the nearest config file (
package.json,go.mod, etc.) and treat that directory as the language root. - Example:
package.jsonat repo root +go.modinservices/api/-> run TS setup at root, Go setup scoped toservices/api/. - Each language gets independent detection, installation, and output. They do not share test directories or configs.
A1b. Framework Detection
After detecting the language, inspect dependency declarations for framework-specific packages. The detected framework determines which test patterns the template test will demonstrate.
TypeScript/JavaScript (check dependencies + devDependencies in package.json):
| Dependency | Framework | Template test approach |
|---|---|---|
express |
Express | supertest route tests |
next |
Next.js | Route handler tests, API route tests |
hono |
Hono | Hono test client |
fastify |
Fastify | app.inject() tests |
| None matched | Generic | Export/function-level unit tests |
Python (check pyproject.toml [project.dependencies] or requirements.txt):
| Dependency | Framework | Template test approach |
|---|---|---|
fastapi |
FastAPI | TestClient, dependency overrides |
django |
Django | TestCase, Client, model tests |
flask |
Flask | Test client, route tests |
typer |
Typer (CLI) | CliRunner, exit codes, output assertions |
click |
Click (CLI) | CliRunner, exit codes, output assertions |
| None matched | Generic | Module/function-level tests |
Go (check require block in go.mod):
| Dependency | Framework | Template test approach |
|---|---|---|
github.com/gin-gonic/gin |
Gin | httptest + gin test context |
github.com/go-chi/chi |
Chi | httptest + chi router |
net/http imports in .go source files (not in go.mod — stdlib packages don't appear there) |
Stdlib | httptest handler tests |
| None matched | Generic | Table-driven function tests |
Rust (check [dependencies] in Cargo.toml):
| Dependency | Framework | Template test approach |
|---|---|---|
actix-web |
Actix | actix_web::test, TestRequest |
axum |
Axum | Tower service tests |
| None matched | Generic | #[cfg(test)] module tests |
Swift (check Package.swift dependencies or project structure):
| Signal | Framework | Template test approach |
|---|---|---|
import Testing in source files (Xcode 16+ / Swift 6) |
Swift Testing | @Test functions, #expect assertions (note: Phase 2/3 templates use XCTest as fallback until Swift Testing templates are added) |
SwiftUI imports + *.xcodeproj dir |
SwiftUI app | ViewInspector, @Observable state tests |
Package.swift (library) |
Swift package | XCTest module tests |
| Vapor in dependencies | Vapor | XCTVapor request tests |
A1c. Existing Test Detection
Before installing, check if test infrastructure already exists for each detected language. Skip or fill gaps as needed.
Signals to check:
| Signal | Means |
|---|---|
Test directories (__tests__/, tests/, test/, *_test.go files) |
Tests may exist |
Test config files (vitest.config.*, jest.config.*, pytest.ini, pyproject.toml with [tool.pytest]) |
Test framework configured |
Test scripts in package.json ("test", "test:coverage") or Makefile (test: target) |
Test runner registered |
Coverage config (.coveragerc, .nycrc, codecov.yml) |
Coverage already set up |
Decision rules:
| Config exists | Test dir exists | Test script exists | Action |
|---|---|---|---|
| Yes | Yes | Yes | Skip -- fully set up |
| Yes | No | -- | Create directory only, keep existing config |
| No | Yes | -- | Create config only, keep existing directory |
| -- | -- | No (but config + dir exist) | Add script only |
| No | No | No | Full setup |
Proceed automatically in all cases (no user prompt). Report what was created vs. what was skipped.
A2. Installation
Install the test framework and coverage provider for each detected language. If installation fails (network, permissions, version conflict), stop and report the error -- do not proceed to Phase 3.
Package Manager Detection (TypeScript/JavaScript)
Detect the package manager from the lock file. Fall back to npm.
| Lock File | Package Manager | Install Command |
|---|---|---|
bun.lockb or bun.lock |
bun | bun add -D vitest @vitest/coverage-v8 |
pnpm-lock.yaml |
pnpm | pnpm add -D vitest @vitest/coverage-v8 |
yarn.lock |
yarn | yarn add -D vitest @vitest/coverage-v8 |
package-lock.json or none |
npm | npm install -D vitest @vitest/coverage-v8 |
Per-Language Installation
TypeScript/JavaScript
- Install vitest + coverage provider via detected package manager
- Install framework-specific test helpers based on detected framework:
| Framework | Additional dev dependency |
|---|---|
| Express | supertest |
| Hono | (built-in test client, no extra dep) |
| Fastify | (built-in app.inject(), no extra dep) |
| Next.js | (no extra dep for route handler tests) |
| Generic | (no extra dep) |
- Create
vitest.config.ts:
import { defineConfig } from 'vitest/config'
export default defineConfig({
test: {
coverage: {
provider: 'v8',
reporter: ['text', 'lcov'], // lcov for Codecov compatibility
exclude: ['node_modules/', 'dist/', '**/*.config.*'],
},
},
})
- Add to
package.jsonscripts:"test": "vitest run""test:coverage": "vitest run --coverage"
- Create
__tests__/directory
Python
- Determine installation method:
uv.lockpresent -> addpytestandpytest-covto dev dependencies, runuv sync --devoruv pip install -e ".[dev]"pyproject.tomlwith PEP 621[project]section -> addpytestandpytest-covto[project.optional-dependencies]dev group, runpip install -e ".[dev]"pyproject.tomlwith Poetry ([tool.poetry]), PDM, or other non-PEP-621 format -> fall back torequirements-dev.txtapproach- No
pyproject.toml-> createrequirements-dev.txtwithpytestandpytest-cov, runpip install -r requirements-dev.txt
- If
$VIRTUAL_ENVis unset and nouv.lock, warn: "No virtual environment detected. Considerpython -m venv .venvfirst." Proceed anyway. - Add pytest config to
pyproject.toml(create or append):
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "--cov=<package> --cov-report=xml --cov-report=term" # Replace <package> with actual package name (e.g., src, app)
- Create
tests/directory with__init__.pyandconftest.py
Go
- No installation needed (testing is built-in)
- If
Makefileexists, add targets:
test:
go test ./...
test-coverage:
go test -coverprofile=coverage.out ./... && go tool cover -html=coverage.out -o coverage.html
- No separate test directory -- Go test files go alongside source files (
*_test.go)
Rust
- No test framework installation needed (built-in
#[test]) - Attempt coverage tool install:
If install fails, warn: "cargo-llvm-cov not installed. Tests will work but coverage reports require it. Install manually or use CI-only coverage." Continue with setup.cargo install cargo-llvm-cov - Create
tests/directory for integration tests
Swift
- XCTest is built-in -- no installation needed
- For
Package.swiftprojects: add test target if missing:.testTarget(name: "AppTests", dependencies: ["App"]) - For Xcode projects: verify test target exists, warn if missing (cannot auto-create Xcode test targets reliably)
- Create
Tests/AppTests/directory structure
A3. Output Files
Generate up to three test files per detected language:
- Smoke test (always) — proves the app can be imported.
- Gold-standard template test (always) — heavily commented pattern for example-based tests.
- Property test template (opt-in) — ask the user first: "Does this repo have parsers, validators, serializers, crypto, state machines, or financial logic?" If yes, generate; if no, skip and don't install the framework.
A. Smoke Test
Generate one real, runnable test that proves the app can be imported without crashing.
Scope: Import-only. Does NOT start servers, connect to databases, or trigger side effects. If the app performs side effects on import (e.g., mongoose.connect() at module level), the smoke test will fail -- report this with the suggestion: "Your app performs side effects on import. Consider wrapping startup logic in a function."
File placement:
| Language | Smoke test file |
|---|---|
| TypeScript/JS | __tests__/smoke.test.ts |
| Python | tests/test_smoke.py |
| Go | smoke_test.go (in root package) |
| Rust | tests/smoke.rs |
| Swift | Tests/AppTests/SmokeTests.swift |
Templates:
import { describe, it, expect } from 'vitest'
describe('smoke', () => {
it('main module imports without error', async () => {
const mod = await import('../src/index')
expect(mod).toBeDefined()
})
})
Adjust the import path (../src/index) to match the actual entry point found in package.json "main" or "exports" field.
def test_smoke():
"""Verify the main package can be imported."""
import app # noqa: F401
Adjust import app to match the actual package name (the top-level directory containing __init__.py, or the module name from pyproject.toml).
package main
import "testing"
func TestSmoke(t *testing.T) {
// Verify the package compiles and main symbols are accessible.
// If this test fails, the package has a build error.
t.Log("smoke test: package compiles successfully")
}
Place in the root package directory. Adjust package main to match the actual package name if different.
#[test]
fn smoke() {
// Verify the crate compiles and can be used as a dependency.
// If this fails, there is a build error in the main crate.
assert!(true, "crate compiles successfully");
}
Place as tests/smoke.rs (integration test). The crate name is auto-resolved from Cargo.toml.
import XCTest
@testable import App
final class SmokeTests: XCTestCase {
func testSmoke() {
// Verify the module can be imported without error.
XCTAssertTrue(true, "Module imports successfully")
}
}
Adjust @testable import App to match the actual module/target name from Package.swift or the Xcode project.
B. Gold-Standard Template Test
Generate one heavily commented test file showing the right patterns for the detected language+framework. Contains 2-3 real implemented tests (not TODOs) demonstrating:
- Happy path -- basic operation with expected input
- Error case -- how to test error handling
- Framework pattern -- one idiomatic framework-specific test (e.g., authenticated route, middleware)
Comments explain: import conventions, test structure, mocking approach, and where to find more patterns.
File placement:
| Language | Template test file | Naming rationale |
|---|---|---|
| TypeScript/JS | __tests__/_template.test.ts |
Underscore sorts first |
| Python | tests/test_template.py |
Follows pytest test_ convention |
| Go | template_test.go (root package) |
Matches template naming in other languages (example_test.go is reserved for godoc examples) |
| Rust | tests/template.rs |
Integration test in tests/ |
| Swift | Tests/AppTests/TemplateTests.swift |
XCTest naming convention |
Generate the template based on the detected framework. Use the framework detection from Phase 1b to select the right test patterns. The template must use the actual framework's test helpers (e.g., supertest for Express, TestClient for FastAPI, httptest for Go stdlib).
References to include in template comments:
busdriver:tdd-- for generating tests for specific modules- Language-specific testing skill --
busdriver:golang-testing,busdriver:python-testing,busdriver:rust-testing, etc.
/**
* TEMPLATE TEST -- Copy this file as a starting point for new test files.
*
* Pattern: supertest + vitest for Express route testing.
* Run: npm test
* Coverage: npm run test:coverage
*
* For full TDD workflow, use `busdriver:tdd` to generate tests for specific modules.
* For more patterns, see `busdriver:tdd`.
*/
import { describe, it, expect } from 'vitest'
import request from 'supertest'
import { app } from '../src/app'
describe('GET /health', () => {
// Happy path: verify the endpoint returns expected shape
it('returns 200 with status ok', async () => {
const res = await request(app).get('/health')
expect(res.status).toBe(200)
expect(res.body).toEqual({ status: 'ok' })
})
// Error case: verify proper error response format
it('returns 404 for unknown routes', async () => {
const res = await request(app).get('/nonexistent')
expect(res.status).toBe(404)
})
// Framework pattern: testing with auth header
it('authenticated route returns 401 without token', async () => {
const res = await request(app).get('/api/protected')
expect(res.status).toBe(401)
})
})
/**
* TEMPLATE TEST -- Copy this file as a starting point for new test files.
*
* Pattern: vitest for unit testing exported functions.
* Run: npm test
* Coverage: npm run test:coverage
*
* For full TDD workflow, use `busdriver:tdd`.
*/
import { describe, it, expect } from 'vitest'
// import { yourFunction } from '../src/utils'
describe('yourFunction', () => {
// Happy path
it.todo('returns expected result for valid input')
// it('returns expected result for valid input', () => {
// const result = yourFunction('valid')
// expect(result).toBe(expected)
// })
// Error case
it.todo('throws on invalid input')
// it('throws on invalid input', () => {
// expect(() => yourFunction(null)).toThrow()
// })
})
"""
TEMPLATE TEST -- Copy this file as a starting point for new test files.
Pattern: pytest + TestClient for FastAPI endpoint testing.
Run: pytest
Coverage: pytest --cov
For full TDD workflow, use `busdriver:tdd`.
For more patterns, see `busdriver:python-testing`.
"""
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
# Happy path: verify endpoint returns expected shape
def test_health_returns_ok():
response = client.get("/health")
assert response.status_code == 200
assert response.json() == {"status": "ok"}
# Error case: verify proper error response
def test_unknown_route_returns_404():
response = client.get("/nonexistent")
assert response.status_code == 404
# Framework pattern: dependency override for testing
def test_with_dependency_override():
"""Example of overriding a FastAPI dependency for testing."""
# from app.dependencies import get_db
# def mock_db():
# return FakeDB()
# app.dependency_overrides[get_db] = mock_db
# response = client.get("/items")
# app.dependency_overrides.clear()
pass # Replace with real test
"""
TEMPLATE TEST -- Copy this file as a starting point for new test files.
Pattern: pytest for unit testing functions and classes.
Run: pytest
Coverage: pytest --cov
For full TDD workflow, use `busdriver:tdd`.
For more patterns, see `busdriver:python-testing`.
"""
# from your_module import your_function
# Happy path: verify function returns expected result
def test_happy_path():
# result = your_function("valid input")
# assert result == expected
pass # Replace with real test
# Error case: verify error handling
def test_error_case():
# with pytest.raises(ValueError):
# your_function(None)
pass # Replace with real test
"""
TEMPLATE TEST -- Copy this file as a starting point for new test files.
Pattern: pytest + Typer's CliRunner for CLI command testing.
Run: pytest
Coverage: pytest --cov
For full TDD workflow, use `busdriver:tdd`.
For more patterns, see `busdriver:python-testing`.
"""
from typer.testing import CliRunner
from app.main import app # Adjust to your Typer app import
runner = CliRunner()
# Happy path: verify command runs and produces expected output
def test_command_succeeds():
result = runner.invoke(app, ["--help"])
assert result.exit_code == 0
assert "Usage" in result.output
# Error case: verify proper error exit on bad input
def test_command_bad_input():
result = runner.invoke(app, ["nonexistent-command"])
assert result.exit_code != 0
# Framework pattern: test a subcommand with arguments
def test_subcommand_with_args(tmp_path):
"""Use tmp_path for any file I/O to keep tests isolated."""
# result = runner.invoke(app, ["process", "--input", str(tmp_path / "data.csv")])
# assert result.exit_code == 0
# assert "Processed" in result.output
pass # Replace with real test
"""
TEMPLATE TEST -- Copy this file as a starting point for new test files.
Pattern: pytest + Click's CliRunner for CLI command testing.
Run: pytest
Coverage: pytest --cov
For full TDD workflow, use `busdriver:tdd`.
For more patterns, see `busdriver:python-testing`.
"""
from click.testing import CliRunner
from app.main import cli # Adjust to your Click group/command import
runner = CliRunner()
# Happy path: verify command runs and produces expected output
def test_command_succeeds():
result = runner.invoke(cli, ["--help"])
assert result.exit_code == 0
assert "Usage" in result.output
# Error case: verify proper error exit on bad input
def test_command_missing_required():
result = runner.invoke(cli, ["process"]) # Missing required arg
assert result.exit_code != 0
assert "Error" in result.output or "Missing" in result.output
# Framework pattern: test with isolated filesystem
def test_command_with_files(tmp_path):
"""Use tmp_path for file I/O; use runner.isolated_filesystem() for CWD isolation."""
# with runner.isolated_filesystem(temp_dir=tmp_path):
# result = runner.invoke(cli, ["init"])
# assert result.exit_code == 0
pass # Replace with real test
// Template test -- copy this file as a starting point for new test files.
//
// Pattern: table-driven tests with httptest for HTTP handler testing.
// Run: make test (or go test ./...)
// Coverage: make test-coverage
//
// For full TDD workflow, use `busdriver:tdd`.
// For more patterns, see `busdriver:golang-testing`.
package main
import (
"net/http"
"net/http/httptest"
"testing"
)
// Happy path: verify handler returns expected status.
func TestHealthHandler(t *testing.T) {
req := httptest.NewRequest(http.MethodGet, "/health", nil)
w := httptest.NewRecorder()
healthHandler(w, req)
if w.Code != http.StatusOK {
t.Errorf("expected 200, got %d", w.Code)
}
}
// Error case: table-driven test pattern for multiple inputs.
func TestHealthHandler_EdgeCases(t *testing.T) {
tests := []struct {
name string
method string
want int
}{
{"GET returns 200", http.MethodGet, http.StatusOK},
{"POST returns 405", http.MethodPost, http.StatusMethodNotAllowed},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req := httptest.NewRequest(tt.method, "/health", nil)
w := httptest.NewRecorder()
healthHandler(w, req)
if w.Code != tt.want {
t.Errorf("expected %d, got %d", tt.want, w.Code)
}
})
}
}
// Template test -- copy this file as a starting point for new test files.
//
// Pattern: table-driven tests for pure functions.
// Run: go test ./...
// Coverage: go test -coverprofile=coverage.out ./...
//
// For full TDD workflow, use `busdriver:tdd`.
// For more patterns, see `busdriver:golang-testing`.
package main
import "testing"
// Happy path: verify function returns expected result.
func TestYourFunction(t *testing.T) {
// result := YourFunction("valid input")
// if result != expected {
// t.Errorf("expected %v, got %v", expected, result)
// }
t.Log("Replace with real test")
}
// Error case: table-driven test pattern.
func TestYourFunction_EdgeCases(t *testing.T) {
tests := []struct {
name string
input string
wantErr bool
}{
{"valid input", "hello", false},
{"empty input", "", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// _, err := YourFunction(tt.input)
// if (err != nil) != tt.wantErr {
// t.Errorf("wantErr=%v, got err=%v", tt.wantErr, err)
// }
t.Log("Replace with real test")
})
}
}
//! TEMPLATE TEST -- Copy this file as a starting point for new integration tests.
//!
//! Pattern: integration test in tests/ directory.
//! Run: cargo test
//! Coverage: cargo llvm-cov
//!
//! For full TDD workflow, use `busdriver:tdd`.
//! For more patterns, see `busdriver:rust-testing`.
// use your_crate::your_function;
// Happy path: verify function returns expected result
#[test]
fn test_happy_path() {
// let result = your_function("valid input");
// assert_eq!(result, expected);
}
// Error case: verify error handling
#[test]
fn test_error_case() {
// let result = your_function("");
// assert!(result.is_err());
}
/// TEMPLATE TEST -- Copy this file as a starting point for new test files.
///
/// Pattern: XCTest for unit testing.
/// Run: swift test (SPM) or xcodebuild test (Xcode)
///
/// For full TDD workflow, use `busdriver:tdd`.
import XCTest
@testable import App
final class TemplateTests: XCTestCase {
// Happy path: verify function returns expected result
func testHappyPath() throws {
// let result = yourFunction("valid")
// XCTAssertEqual(result, expected)
throw XCTSkip("Template — replace with real test")
}
// Error case: verify error handling
func testErrorCase() throws {
// XCTAssertThrowsError(try yourFunction(nil))
throw XCTSkip("Template — replace with real test")
}
}
Adapt all import paths and function names to match the actual codebase. The template is a starting point -- the tests should compile and pass as-is, so the developer can immediately see the pattern and replace with real tests.
Placeholder tests: Avoid always-passing no-op assertions (assert True, expect(true).toBe(true), XCTAssertTrue(true)) in template tests -- they inflate pass counts and trigger automated reviewer warnings. Instead:
- Python: Use
passfor placeholder bodies - TypeScript/JS: Use
it.todo('description')(vitest/jest mark them as pending, not passing) - Swift: Use
throw XCTSkip("Template — replace with real test") - Go:
t.Log(...)is fine (informational, not a false assertion) - Rust: Commented-out assertions are fine (no placeholder needed)
For tests that demonstrate a real pattern (e.g., importing the app, hitting a real endpoint), use actual assertions -- only use placeholders for commented-out examples the developer hasn't wired up yet.
C. Property-Based Test Template (Optional)
Property-based testing complements example-based testing by generating random inputs and checking that invariants hold across all of them. It catches edge cases that example tests miss (empty strings, unicode boundaries, integer overflow, concurrent-operation orderings).
When to use property tests:
| Good fit | Poor fit |
|---|---|
Parsers / serializers — round-trip invariants (parse(serialize(x)) == x) |
CRUD endpoints / HTTP glue |
| Validators — rejected inputs stay rejected after canonicalization | UI rendering logic |
| Crypto / hashing — output length, determinism, collision properties | Database migrations |
| State machines — sequences of operations preserve invariants | Configuration loaders |
| Financial calculators — commutative / associative / zero-sum properties | Pure plumbing / pass-throughs |
| Sort / search / data structures — algorithmic invariants | Pure presentation-layer code |
If a repo has no modules matching "good fit," skip this template — adding property tests to CRUD handlers is noise, not signal.
Three gold-standard property patterns each template demonstrates:
- Invariant — a property that always holds (
reverse(reverse(x)) == x,parse(serialize(x)) == x) - Oracle — compare new implementation against a simple reference (
my_sort(xs) == sorted(xs)) - Model/state-machine — a sequence of operations preserves a higher-level invariant (
push then pop on stack returns same element)
File placement:
| Language | Property test file | Framework | Install |
|---|---|---|---|
| Python | tests/test_properties_template.py |
Hypothesis | pip install hypothesis |
| TypeScript/JS | __tests__/properties.test.ts |
fast-check | npm install -D fast-check |
| Go | properties_test.go |
rapid (preferred — has shrinking) or testing/quick (stdlib, no shrinking) |
go get -t pgregory.net/rapid |
| Rust | tests/properties.rs |
proptest | cargo add --dev proptest |
| Swift | Tests/AppTests/PropertyTests.swift |
Pragmatic mix — see Swift template notes | See notes |
Opt-in flow: During Phase A, after detecting language + modules, ask the user:
"Does this repo have parsers, validators, serializers, crypto, state machines, or financial logic? (y/n)"
If yes → generate property test template. If no → skip; add no framework dependency.
Default is no — don't install framework dependencies speculatively.
"""
PROPERTY TEST TEMPLATE -- Copy this file as a starting point.
When to use: parsers, serializers, validators, crypto, state machines,
financial calculators. NOT for CRUD, HTTP glue, or pure plumbing.
Pattern: Hypothesis generates random inputs; you assert invariants.
Run: pytest tests/test_properties_template.py
Install: pip install hypothesis
For more patterns, see https://hypothesis.readthedocs.io/en/latest/quickstart.html
"""
import pytest
from hypothesis import given, strategies as st
# from your_module import serialize, deserialize, validate, canonicalize
# 1. INVARIANT -- round-trip property: parse(serialize(x)) == x
@given(st.dictionaries(st.text(), st.integers()))
def test_serialize_roundtrip_is_identity(data):
"""Serializing then parsing should always return the original."""
pytest.skip("Template — replace body with real serialize/deserialize round-trip")
# assert deserialize(serialize(data)) == data
# 2. ORACLE -- compare implementation to a known-correct reference
@given(st.lists(st.integers()))
def test_custom_sort_matches_python_sorted(xs):
"""Your sort should match Python's built-in sorted()."""
pytest.skip("Template — replace body with real my_sort vs sorted() comparison")
# assert my_sort(list(xs)) == sorted(xs)
# 3. MODEL/STATE-MACHINE -- sequences of ops preserve an invariant
@given(st.lists(st.integers()))
def test_push_then_pop_returns_same_element(items):
"""Stack push/pop is a two-way mapping for each element."""
pytest.skip("Template — replace body with real stack LIFO check")
# stack = Stack()
# for item in items:
# stack.push(item)
# assert stack.pop() == item
# Shrinking example -- Hypothesis will minimize a failing input.
# Uncomment to see: the test will "fail" with a minimal counterexample.
# @given(st.lists(st.integers()))
# def test_intentionally_fails_to_show_shrinking(xs):
# assert sum(xs) < 1_000_000 # Shrinker finds [1_000_000] as minimal fail
/**
* PROPERTY TEST TEMPLATE -- Copy this file as a starting point.
*
* When to use: parsers, serializers, validators, crypto, state machines,
* financial calculators. NOT for CRUD, HTTP glue, or pure plumbing.
*
* Pattern: fast-check generates random inputs; you assert invariants.
* Run: npm test -- properties
* Install: npm install -D fast-check
*
* For more patterns, see https://fast-check.dev/docs/tutorials/quick-start/
*/
import { describe, it } from 'vitest'
// import fc from 'fast-check'
// import { serialize, deserialize, mySort, Stack } from '../src/module'
describe('property tests', () => {
// 1. INVARIANT -- round-trip property
it.todo('serialize + deserialize is identity')
// it('serialize + deserialize is identity', () => {
// fc.assert(
// fc.property(fc.dictionary(fc.string(), fc.integer()), (data) => {
// expect(deserialize(serialize(data))).toEqual(data)
// })
// )
// })
// 2. ORACLE -- compare to a reference implementation
it.todo('custom sort matches Array.prototype.sort')
// it('custom sort matches Array.prototype.sort', () => {
// fc.assert(
// fc.property(fc.array(fc.integer()), (xs) => {
// const mine = mySort([...xs])
// const reference = [...xs].sort((a, b) => a - b)
// expect(mine).toEqual(reference)
// })
// )
// })
// 3. MODEL/STATE-MACHINE -- sequence of ops preserves invariant
it.todo('stack push/pop preserves last-in-first-out')
// it('stack push/pop preserves last-in-first-out', () => {
// fc.assert(
// fc.property(fc.array(fc.integer()), (items) => {
// const stack = new Stack<number>()
// for (const item of items) {
// stack.push(item)
// expect(stack.pop()).toBe(item)
// }
// })
// )
// })
// Shrinking example -- fast-check minimizes a failing input.
// Uncomment to see: failure report will show the minimal counterexample.
// it('intentionally fails to demonstrate shrinking', () => {
// fc.assert(
// fc.property(fc.array(fc.integer()), (xs) => {
// expect(xs.reduce((a, b) => a + b, 0)).toBeLessThan(1_000_000)
// })
// )
// })
})
Recommended: pgregory.net/rapid. It provides shrinking (automatic minimization of failing inputs), stateful testing, and rich generators. testing/quick works but lacks shrinking — debugging failures on nested structs becomes painful.
// Package myapp_test contains property-based tests using pgregory.net/rapid.
//
// When to use: parsers, serializers, validators, crypto, state machines,
// financial calculators. NOT for CRUD, HTTP glue, or pure plumbing.
//
// Pattern: rapid generates random inputs and SHRINKS failures to minimal cases.
// Run: go test -run TestProperty ./...
// Install: go get -t pgregory.net/rapid
//
// For more patterns, see https://pkg.go.dev/pgregory.net/rapid
// Fallback (stdlib, no shrinking): use testing/quick — see commented section below.
package myapp_test
import (
"testing"
// Uncomment when you replace t.Skip() bodies with real property checks:
// "sort"
// "pgregory.net/rapid"
)
// 1. INVARIANT -- round-trip property
func TestPropertySerializeRoundtrip(t *testing.T) {
t.Skip("template — replace body with real serialize/deserialize round-trip check")
// rapid.Check(t, func(t *rapid.T) {
// data := rapid.MapOf(rapid.String(), rapid.Int()).Draw(t, "data")
// encoded := serialize(data)
// decoded := deserialize(encoded)
// if !reflect.DeepEqual(decoded, data) {
// t.Fatalf("roundtrip mismatch: got %v, want %v", decoded, data)
// }
// })
}
// 2. ORACLE -- compare custom impl against sort.Ints
func TestPropertyCustomSortMatchesStdlib(t *testing.T) {
t.Skip("template — replace body with real mySort vs sort.Ints comparison")
// rapid.Check(t, func(t *rapid.T) {
// xs := rapid.SliceOf(rapid.Int()).Draw(t, "xs")
// mine := mySort(append([]int{}, xs...))
// reference := append([]int{}, xs...)
// sort.Ints(reference)
// if !slices.Equal(mine, reference) {
// t.Fatalf("sort mismatch: got %v, want %v", mine, reference)
// }
// })
}
// 3. MODEL/STATE-MACHINE -- stack push/pop invariant
func TestPropertyStackPushPopLIFO(t *testing.T) {
t.Skip("template — replace body with real stack LIFO invariant check")
// rapid.Check(t, func(t *rapid.T) {
// items := rapid.SliceOf(rapid.Int()).Draw(t, "items")
// stack := NewStack[int]()
// for _, item := range items {
// stack.Push(item)
// if got := stack.Pop(); got != item {
// t.Fatalf("LIFO violated: push %d then pop %d", item, got)
// }
// }
// })
}
// ── Stdlib fallback (no shrinking) ───────────────────────────────────────────
// If you cannot add rapid as a dependency, testing/quick from the stdlib works.
// Failure reports show raw generated inputs (no minimization), so debugging is
// harder — especially for maps and nested structs.
//
// import "testing/quick"
//
// func TestPropertyQuickSerializeRoundtrip(t *testing.T) {
// f := func(data map[string]int) bool {
// return reflect.DeepEqual(deserialize(serialize(data)), data)
// }
// if err := quick.Check(f, nil); err != nil {
// t.Error(err)
// }
// }
//! PROPERTY TEST TEMPLATE -- Copy this file as a starting point.
//!
//! When to use: parsers, serializers, validators, crypto, state machines,
//! financial calculators. NOT for CRUD, HTTP glue, or pure plumbing.
//!
//! Pattern: proptest generates random inputs; you assert invariants.
//! Run: cargo test --test properties
//! Install: cargo add --dev proptest
//!
//! For more patterns, see https://proptest-rs.github.io/proptest/
use proptest::prelude::*;
// use std::collections::HashMap; // uncomment when you wire up the serialize test
// use my_crate::{serialize, deserialize, my_sort};
proptest! {
// 1. INVARIANT -- round-trip property
// Remove #[ignore] once you wire up real serialize/deserialize.
#[test]
#[ignore = "template — replace body with real round-trip check"]
fn serialize_roundtrip_is_identity(_data in prop::collection::hash_map(".*", any::<i64>(), 0..10)) {
// let encoded = serialize(&_data);
// let decoded: HashMap<String, i64> = deserialize(&encoded).unwrap();
// prop_assert_eq!(decoded, _data);
}
// 2. ORACLE -- compare custom impl against stdlib sort
#[test]
#[ignore = "template — replace body with real my_sort vs stdlib comparison"]
fn custom_sort_matches_stdlib(_xs in prop::collection::vec(any::<i32>(), 0..100)) {
// let mut mine = _xs.clone();
…(truncated)