Test Fixture Creation

Test Fixture Creation

Guidelines for creating test fixtures that work with Bulwark's hook-based automation and avoid bias that could compromise testing.

When to Use This Skill

Load this skill when:

• Creating fixtures for manual testing of skills or agents
• Setting up E2E test scenarios that require hook automation
• Building code samples with deliberate issues for LLM review

DO NOT use for:

• Unit test fixtures (those can be isolated in tests/fixtures/)
• Mock data for automated tests
• Documentation examples

Core Principles

1. No Bias in Fixtures

CRITICAL: Fixtures must not contain any indicators that they are test fixtures.

Why this matters: When Claude knows code is a test fixture, it may:

• Skip hook automation ("this is just a test")
• Ignore pipeline suggestions
• Produce different results than real code review

2. Project Infrastructure Integration

Fixtures must be placed within project infrastructure to enable hook automation.

Required for hooks to fire:

• Code must be in directories covered by tsconfig.json include paths
• Project must have working just typecheck and just lint recipes
• Fixtures must pass Phase 1 checks

Placement Strategy:

PROJECT_ROOT/
├── scripts/
│   ├── components/     ← Place fixtures here
│   │   ├── user-service.ts
│   │   ├── data-processor.ts
│   │   └── workflow-handler.ts
│   └── lib/            ← Supporting stubs
│       ├── database.ts
│       └── logger.ts

3. Fixtures Must Pass Phase 1

Fixtures should compile and lint successfully so that Phase 2 (LLM review) can run.

Phase 1 Requirements:

• just typecheck passes (no TypeScript errors)
• just lint passes (no lint errors)
• All imports resolve

Common Issues:

4. Deliberate Issues for Phase 2

Fixtures should contain issues that TypeScript allows but are bad practice:

Security Issues (user-service.ts):

• SQL injection via string interpolation
• Hardcoded API keys and secrets
• Path traversal vulnerabilities
• Insecure token generation

Type Safety Issues (data-processor.ts):

• Excessive any in properties and parameters
• Unsafe type assertions (as unknown as T, as any)
• Missing return types

Linting Issues (workflow-handler.ts):

• Single-letter function names (p, x, z)
• Generic variable names (s, d, c, i, r)
• Deep nesting (8+ levels)
• High cyclomatic complexity

Coding Standards Issues (config-manager.ts):

• Multiple responsibilities in one file
• Global mutable state
• Implicit side effects (auto-initialization)
• Mixed concerns in functions

Fixture Creation Workflow

Step 1: Plan Fixture Structure

1. Identify skill/agent sections to test
2. Map each section to a fixture file
3. Plan deliberate issues for each file
4. Identify supporting stubs needed

Step 2: Create Supporting Infrastructure

// scripts/lib/database.ts - Stub for database imports
export interface QueryResult {
  rows: Record<string, unknown>[];
}

export const db = {
  async query(sql: string): Promise<QueryResult> {
    return { rows: [] };
  }
};

Step 3: Create Fixture Files

// scripts/components/user-service.ts
// NO comments explaining this is a test!
// File looks like production code

import { db } from '../lib/database';

const API_KEY = 'sk_live_...';  // Hardcoded secret

export async function getUserByEmail(email: string) {
  const query = `SELECT * FROM users WHERE email = '${email}'`;  // SQL injection
  // ... rest of realistic code
}

Step 4: Verify Phase 1 Passes

# Must pass before creating test cases
just typecheck
just lint

Step 5: Create Test Protocol

Use conversational, non-developer prompts:

**Prompt** (conversational):
I just joined the team and was asked to review the user authentication module
before we go live. Can you take a look at scripts/components/user-service.ts
and let me know if there's anything concerning?

NOT:

**Prompt** (too technical):
Run code-review on the SQL injection vulnerability in getUserByEmail().

Step 6: Add Cleanup Steps

## Cleanup Steps

### CLEANUP-P4-001: Remove Component Fixtures
rm -rf scripts/components/
rm -rf scripts/lib/database.ts
rm -rf scripts/lib/logger.ts

Prompt Writing Guidelines

Realistic User Language

Users don't speak in technical jargon. Prompts should reflect real conversations.

Symptom-Based Prompts

Describe symptoms, not root causes:

"After I select a date range on the app, the app hangs. Could you please
debug, fix and validate the issue after loading the appropriate skills
and pipeline agents?"

NOT:

"timerangecalc() is giving an out of bound error"

Cleanup Protocol

MANDATORY: All fixtures must be cleaned up after testing.

Cleanup Checklist

1. Remove fixture files (scripts/components/*.ts)
2. Remove supporting stubs (scripts/lib/database.ts, etc.)
3. Remove empty directories
4. Clear diagnostic logs (optional)
5. Verify just typecheck passes after cleanup
6. Check git status for orphaned artifacts

Cleanup Commands

# Remove fixtures
rm -rf scripts/components/
rm -f scripts/lib/database.ts scripts/lib/logger.ts

# Verify clean state
just typecheck
git status

Diagnostic Output

After creating fixtures, document the mapping:

diagnostic:
  skill: test-fixture-creation
  timestamp: 2026-01-31T12:00:00Z
  fixtures_created:
    - file: scripts/components/user-service.ts
      section: Security
      issues: [sql_injection, hardcoded_secrets, path_traversal]
    - file: scripts/components/data-processor.ts
      section: Type Safety
      issues: [excessive_any, unsafe_assertions]
    - file: scripts/components/workflow-handler.ts
      section: Linting
      issues: [poor_naming, deep_nesting, high_complexity]
    - file: scripts/components/config-manager.ts
      section: Coding Standards
      issues: [multiple_responsibilities, global_state, side_effects]
  supporting_stubs:
    - scripts/lib/database.ts
    - scripts/lib/logger.ts
  phase1_validation:
    typecheck: passed
    lint: passed

Learnings Log

Session 32 (2026-01-31)

1. Fixtures in project infrastructure: Test fixtures need to be in project directories (not tests/fixtures/) so hooks fire correctly
2. No @types/node: Pure TypeScript fixtures avoid dependency on @types/node
3. Stub imports: Create minimal stubs for imports to make fixtures compile
4. Realistic prompts: Non-developer language produces more realistic test results
5. Cleanup discipline: Always add cleanup steps to test protocols
6. ESLint config required: Without .eslintrc.json, just lint skips TypeScript - Phase 1 incomplete
7. Package.json for deps: Need package.json with typescript, eslint, @typescript-eslint/* for real Phase 1 checks
8. Warnings vs Errors: ESLint warnings (intentional any, unused vars) don't block Phase 1; only errors block
9. Separate fixture locations: Use scripts/components/ for direct invocation, scripts/services/ for pipeline integration
10. Pipeline vs Direct tests: Direct invocation uses --section flag (deterministic); pipeline tests LLM judgment (stochastic)