CFN Test Runner Skill
Version: 1.0.0 Status: Production Purpose: Unified test execution with benchmarking and regression detection
Overview
Provides comprehensive test execution across all CFN test suites:
- Hello World Tests (Layer 0-7)
- CFN Loop E2E Tests (9 integration tests)
- Benchmark Tracking (SQLite storage)
- Regression Detection (automated alerts)
Key Features:
- Single command test execution
- Historical benchmark comparison
- Performance regression alerts
- Git-aware baseline tracking
- Parallel test execution
- Comprehensive reporting
Architecture
Skill Components
.claude/skills/cfn-test-framework/lib/runner/
├── SKILL.md # This file
├── run-all-tests.sh # Main test runner
├── store-benchmarks.sh # SQLite benchmark storage
├── detect-regressions.sh # Regression analysis
├── init-benchmark-db.sh # Database initialization
└── generate-report.sh # Test report generation
Usage
Run All Tests
$HOME/.claude/skills/cfn-test-framework/lib/runner/run-all-tests.sh \
--suite all \
--benchmark \
--detect-regressions
Run Specific Suite
# Hello World only (archived 2026-08-20, see "Test Suites" below -- this now
# warns and exits 2 without running; it does not report a result)
$HOME/.claude/skills/cfn-test-framework/lib/runner/run-all-tests.sh --suite hello-world
# CFN E2E only
$HOME/.claude/skills/cfn-test-framework/lib/runner/run-all-tests.sh --suite cfn-e2e
# Both
$HOME/.claude/skills/cfn-test-framework/lib/runner/run-all-tests.sh --suite all
With Regression Detection
$HOME/.claude/skills/cfn-test-framework/lib/runner/run-all-tests.sh \
--suite all \
--benchmark \
--detect-regressions \
--threshold 0.10
Parameters
run-all-tests.sh
| Parameter | Required | Description | Default |
|---|---|---|---|
--suite |
No | Test suite: all, hello-world (archived, warns and exits 2 without running -- see Test Suites below), cfn-e2e |
all |
--benchmark |
No | Store results in SQLite | false |
--detect-regressions |
No | Run regression analysis | false |
--threshold |
No | Regression threshold (%) | 0.10 |
--parallel |
No | Run tests in parallel | false |
--output |
No | Output format: text, json, html |
text |
Test Suites
Hello World (Layer 0-7) -- ARCHIVED, DOES NOT RUN
Purpose: Validate agent spawning patterns and coordination
| Layer | Test | Duration | Critical |
|---|---|---|---|
| 0 | Tool Validation | 60s | ✅ |
| 1-4 | [Future] | - | - |
| 5 | Coordinator Spawning | 120s | ✅ |
| 6 | Review Handoff | 180s | ✅ |
| 7 | Error Retry | 150s | ✅ |
Status (as of 2026-08-20): removed from run-all-tests.sh. tests/hello-world/
no longer exists; the source files were archived wholesale to
tests/archive/legacy-v1/hello-world/ (and a second copy under
tests/archive/experimental/hello-world/). Verified broken from the archive
location, not just relocated: Layer 0 spawns agents using a cwd built from a
relative path sized for the old tests/hello-world/ depth, so it now resolves
to the wrong directory (npm error could not determine executable to run).
Layers 5-7 import { createClient } from 'redis', a package this repo no
longer depends on (migrated to ioredis), so they fail at module-resolution
time before attempting any Redis connection. --suite hello-world now warns
and exits 2 rather than running these. It exits non-zero on purpose: running
zero tests and exiting clean would read to any caller as "the suite passed".
Location: tests/archive/legacy-v1/hello-world/ (archived, not wired into the runner)
CFN Loop E2E (9 Tests)
Purpose: Validate full CFN Loop workflow
| Test | Component | Duration | Critical |
|---|---|---|---|
| 1 | Coordinator → Orchestrator | 30s | ✅ |
| 2 | Loop 3 → Gate Check | 60s | ✅ |
| 3 | Gate Pass → Loop 2 | 30s | ⚠️ |
| 4 | Loop 2 → Product Owner | 90s | ✅ |
| 5 | Product Owner Decision | 60s | ✅ |
| 6 | Iteration Cycle | 120s | ✅ |
| 7 | Redis Key Structure | 10s | ✅ |
| 8 | Error Recovery | 60s | ⚠️ |
| 9 | Cleanup | 10s | ⚠️ |
Location: tests/cfn-v3/test-e2e-cfn-loop.sh
Benchmark Storage
SQLite Schema
test_suites -- Suite definitions
test_runs -- Execution records
test_cases -- Individual tests
test_results -- Per-test outcomes
performance_metrics -- Performance data
regression_alerts -- Detected regressions
Database: .artifacts/test-benchmarks.db
Stored Metrics
Test-Level:
- Duration (ms)
- Status (passed/failed/skipped)
- Assertion count
- Error messages
Suite-Level:
- Total duration (seconds)
- Success rate (%)
- Pass/fail/skip counts
- Git commit/branch
Performance:
- Agent spawn time
- Redis operation latency
- File I/O duration
- Memory usage
Regression Detection
Automatic Alerts
# Critical: Test started failing
alert_type: test_failure
severity: critical
message: "TEST 5: Product Owner Decision FAILED (was passing)"
# Warning: Performance degraded
alert_type: performance_regression
severity: warning
message: "Loop 3 spawn time increased 25% (baseline: 5s, current: 6.25s)"
# Info: Success rate dropped
alert_type: success_rate_drop
severity: info
message: "Suite success rate: 88% → 77% (threshold: 10%)"
Threshold Configuration
# Default: 10% regression threshold
--threshold 0.10
# Strict: 5% threshold
--threshold 0.05
# Relaxed: 20% threshold
--threshold 0.20
Output Formats
Text (Console)
Example from before the Hello World suite was archived (2026-08-20). The
"Suite: Hello World" block below no longer appears in real output --
--suite hello-world now prints a warning and skips (see Test Suites above).
==========================================
CFN Test Suite Results
==========================================
Suite: Hello World
Layer 0: ✅ PASSED (58.3s)
Layer 5: ✅ PASSED (115.7s)
Layer 6: ✅ PASSED (172.4s)
Layer 7: ✅ PASSED (148.2s)
Suite: CFN E2E
TEST 1: ✅ PASSED (28.1s)
TEST 2: ✅ PASSED (54.6s)
TEST 3: ⚠️ SKIPPED
TEST 4: ✅ PASSED (87.3s)
TEST 5: ✅ PASSED (59.2s)
TEST 6: ✅ PASSED (118.9s)
TEST 7: ✅ PASSED (9.4s)
Total: 11 tests, 9 passed, 0 failed, 2 skipped
Duration: 851.1s
Success Rate: 81.8%
Regressions Detected: 0
==========================================
JSON (API/CI)
Same caveat as above: hello-world was archived 2026-08-20 and no longer
produces real results; kept here only as a shape example.
{
"timestamp": "2025-11-04T02:00:00Z",
"git_commit": "abc123",
"git_branch": "main",
"suites": {
"hello-world": {
"total": 4,
"passed": 4,
"failed": 0,
"skipped": 0,
"duration": 494.6,
"success_rate": 1.0
},
"cfn-e2e": {
"total": 9,
"passed": 7,
"failed": 0,
"skipped": 2,
"duration": 356.5,
"success_rate": 0.778
}
},
"regressions": [],
"baseline_comparison": {
"duration_change": "+2.3%",
"success_rate_change": "+0.0%"
}
}
HTML (Report)
Generates interactive HTML report with:
- Trend graphs (Chart.js)
- Regression highlights
- Test history
- Performance metrics
Baseline Management
Establish Baseline
# First run establishes baseline
$HOME/.claude/skills/cfn-test-framework/lib/runner/run-all-tests.sh --suite all --benchmark
# Mark as baseline
sqlite3 .artifacts/test-benchmarks.db << EOF
UPDATE test_runs SET environment = 'baseline' WHERE id = (SELECT MAX(id) FROM test_runs);