2319 Guide 02 Building Your Own 5655185e

GUIDE 02: Building Your Own Workflow

Read time: 30 minutes Goal: Adapt the test harness pattern for your specific use case

Decision Tree: Do I Need This Pattern?

Start: I have a multi-step workflow

Q1: Does it involve more than 2 steps?
    No → Just use a simple prompt, no need for test harness
    Yes → Continue to Q2

Q2: Do I need to verify the work was actually done?
    No → Simple multi-turn conversation is fine
    Yes → Continue to Q3

Q3: Can I validate outputs programmatically?
    No → Test harness won't help (need human judgment)
    Yes → Continue to Q4

Q4: Is there a deterministic way to check conclusions?
    No → Consider adding one (script/tool), but pattern still useful
    Yes → USE TEST HARNESS PATTERN ✅

Example YES cases:
- Data pipeline validation
- Code review workflows
- Compliance audits
- Research synthesis with citation checking
- Security scans with tool verification

Example NO cases:
- Creative writing (no objective validation)
- Open-ended brainstorming (no right answer)
- Simple Q&A (single-turn)
- Purely subjective evaluations

Step-by-Step: Adapt for Your Use Case

Step 1: Define Your Workflow Phases

Template:

Phase 1: [Initial Analysis/Collection]
Purpose: Establish baseline, gather data
Output: Summary of what was found

Phase 2: [Deep Analysis]
Purpose: Analyze patterns, identify issues
Output: Detailed findings with evidence

Phase 3: [Risk/Impact Assessment]
Purpose: Evaluate consequences, prioritize
Output: Risk scores, impact estimates

Phase 4: [Verification]
Purpose: Empirically validate conclusions
Output: Confirmed vs revised findings

Phase 5: [Recommendations]
Purpose: Synthesize into action plan
Output: Prioritized recommendations

Your workflow:

1. Write down your end goal
2. Work backwards: what's the last step before that goal?
3. What needs to happen before that step?
4. Keep breaking down until you have 3-7 phases
5. Identify which phase can run a deterministic check (your "Phase 4")

Examples:

Code Review Workflow:

Phase 1: File Discovery - Find all changed files
Phase 2: Static Analysis - Run linters/formatters
Phase 3: Pattern Detection - Identify anti-patterns
Phase 4: Test Verification - Run test suite, compare coverage
Phase 5: Review Summary - Aggregate all findings

Security Audit Workflow:

Phase 1: Surface Scan - Identify entry points
Phase 2: Vulnerability Analysis - Check OWASP Top 10
Phase 3: Risk Scoring - Categorize by severity
Phase 4: Tool Verification - Run SAST/DAST, compare findings
Phase 5: Remediation Plan - Prioritize fixes

Research Synthesis Workflow:

Phase 1: Paper Collection - Gather relevant papers
Phase 2: Theme Extraction - Identify common themes
Phase 3: Gap Analysis - Find research gaps
Phase 4: Citation Verification - Run citation analysis, verify connections
Phase 5: Synthesis Report - Write literature review

Step 2: Design Your Session Directory

Pattern:

reports/runs/<timestamp>/
├── 01-<phase1-name>.md
├── 02-<phase2-name>.md
├── 03-<phase3-name>.md
├── 04-<phase4-name>.md
└── 05-<phase5-name>.md

Customization:

• Rename reports/ to match your domain (e.g., audits/, reviews/, syntheses/)
• Add phase-specific artifacts (e.g., 04-test-results.json, 02-lint-output.txt)
• Include metadata file (e.g., _session-metadata.json)

Example: Code Review

reviews/runs/2025-01-15_143022/
├── 01-file-discovery.md
├── 02-static-analysis.md
├── 02-lint-output.txt           # Linter raw output
├── 03-pattern-detection.md
├── 04-test-verification.md
├── 04-test-results.json         # Test suite JSON output
├── 05-review-summary.md
└── _metadata.json               # PR number, author, timestamp

Step 3: Define JSON Contracts

For each phase, specify:

{
  "status": "complete",
  "report_path": "/absolute/path/to/report.md",
  "phase_summary": {
    // Phase-specific keys here
  }
}

Phase 1 example (File Discovery):

{
  "status": "complete",
  "report_path": "/path/to/01-file-discovery.md",
  "phase_summary": {
    "files_changed": 12,
    "lines_added": 456,
    "lines_removed": 123,
    "languages": ["python", "javascript"]
  }
}

Phase 4 example (Test Verification):

{
  "status": "complete",
  "report_path": "/path/to/04-test-verification.md",
  "verification_summary": {
    "tests_run": 145,
    "tests_passed": 143,
    "tests_failed": 2,
    "coverage_before": 78.5,
    "coverage_after": 82.3,
    "manual_predictions_confirmed": [
      "Predicted test_foo would fail: CONFIRMED"
    ],
    "manual_predictions_revised": [
      "Predicted test_bar would pass: REVISED (failed)"
    ]
  }
}

Tip: Phase 4 should always have:

• conclusions_confirmed: What manual analysis got right
• conclusions_revised: What needed correction
• unexpected_findings: What manual analysis missed

Step 4: Write Reference Docs

Template for each reference doc:

# Phase N: [Name]

## Context
[1-2 paragraphs: what this phase does, why it matters]

## Inputs Available

You have access to:
- `skill_dir`: [describe]
- `session_dir`: [describe]
- `input_folder`: [describe]
- `phaseX_report_path`: [describe prior phases]

## Step-by-Step Procedure

### Step 1: [Concrete Action]
[Detailed instructions]
[Example commands/code if applicable]

Expected output: [describe]

### Step 2: [Concrete Action]
[Detailed instructions]

Expected output: [describe]

### Step 3: [Concrete Action]
[Continue until work is complete]

## Report Format

Write to: `{session_dir}/0N-[name].md`

Required sections:
- ## Executive Summary (3-5 bullets)
- ## Methodology (how you did it)
- ## Findings (what you found, with evidence)
- ## [Phase-Specific Section]
- ## Next Steps (what Phase N+1 should focus on)

## Output Requirements

Return JSON only (no explanatory text):
```json
{
  "status": "complete",
  "report_path": "<absolute path>",
  "phase_summary": {
    "key1": "<value>",
    "key2": "<value>"
  }
}

Quality Checklist

• All data sources documented in report
• Metrics include methodology
• Findings are evidence-based (not speculation)
• Report file exists on disk before returning JSON
• JSON includes all required keys

**Critical: Phase 4 Reference (Verification)**

```markdown
# Phase 4: [Name] Verification

## Context
Phases 2-3 made conclusions through manual analysis.
This phase runs deterministic tools to empirically validate those conclusions.

## Key Conclusions from Prior Phases to Verify

Read `phase2_report_path` and `phase3_report_path`.

Extract the following claims:
- [Specific claim type 1]
- [Specific claim type 2]
- [Specific claim type 3]

## Step-by-Step Procedure

### Step 1: Extract Manual Conclusions
From Phase 2/3 reports, create a structured list:

Claim 1: [description] Evidence cited: [what Phase 2/3 based this on]

Claim 2: [description] Evidence cited: [what Phase 2/3 based this on]

### Step 2: Execute Verification Tool
Run: `scripts/[your-verification-script.sh] <input> <output>`

This script must:
- Be deterministic (same inputs = same outputs)
- Produce structured output (JSON preferred)
- Complete quickly (<5 minutes)

### Step 3: Parse Tool Output
Read the script's output file.
Extract measurable facts (not opinions).

### Step 4: Compare Manual vs Empirical
For each claim from Phase 2/3:

If tool confirms (matches within tolerance):
→ Add to `conclusions_confirmed`

If tool differs significantly:
→ Add to `conclusions_revised` with explanation

If tool finds issues not in Phase 2/3:
→ Add to `unexpected_findings`

### Step 5: Update Action Items
Based on comparison:
- Keep actions based on confirmed conclusions
- Revise actions based on revised conclusions
- Add new actions based on unexpected findings

### Step 6: Write Verification Report
Save to: `{session_dir}/04-[name]-verification.md`

Required sections:
- ## Original Conclusions (from Phase 2/3)
- ## Tool Execution Results
- ## Comparison Analysis (table format)
  | Claim | Manual Analysis | Tool Result | Status | Notes |
  |-------|----------------|-------------|--------|-------|
  | ...   | ...            | ...         | ...    | ...   |
- ## Revised Action Items

### Step 7: Return JSON
```json
{
  "status": "complete",
  "report_path": "<absolute>",
  "verification_summary": {
    "tool_executed": "<script name>",
    "tool_runtime_seconds": N,
    "conclusions_confirmed": ["claim1: confirmed within 5% tolerance"],
    "conclusions_revised": ["claim2: was X, actually Y"],
    "unexpected_findings": ["claim3: tool found issue not in manual analysis"],
    "revised_action_items": ["action1: safe (confirmed)", "action2: needs review (revised)"]
  }
}

### Step 5: Create Verification Script

**Requirements:**
1. **Deterministic:** No randomness, no LLM calls, no API dependencies
2. **Fast:** Completes in seconds to minutes (not hours)
3. **Structured output:** JSON preferred, CSV acceptable
4. **Self-validating:** Checks inputs before processing
5. **Error handling:** Returns meaningful errors

**Template:**

```bash
#!/bin/bash
# [Your Verification Script]
# Purpose: [What it verifies]
# Inputs: [Describe]
# Outputs: [Describe]

set -euo pipefail  # Exit on error

# ===========================================
# Configuration
# ===========================================
SCRIPT_NAME=$(basename "$0")
VERSION="1.0.0"

# ===========================================
# Input Validation
# ===========================================
if [ $# -ne 2 ]; then
  echo "Usage: $SCRIPT_NAME <input_path> <output_path>"
  echo "  input_path: Directory or file to analyze"
  echo "  output_path: Where to write results"
  exit 1
fi

INPUT_PATH="$1"
OUTPUT_PATH="$2"

if [ ! -e "$INPUT_PATH" ]; then
  echo "Error: Input path does not exist: $INPUT_PATH"
  exit 1
fi

mkdir -p "$OUTPUT_PATH"

# ===========================================
# Main Analysis Logic
# ===========================================
echo "Starting analysis..."
START_TIME=$(date +%s)

# [Your analysis code here]
# Example: Parse files, calculate metrics, etc.

METRICS_CALCULATED=0
ISSUES_FOUND=0

for file in "$INPUT_PATH"/*; do
  if [ -f "$file" ]; then
    # Process file
    # Increment metrics
    METRICS_CALCULATED=$((METRICS_CALCULATED + 1))
  fi
done

END_TIME=$(date +%s)
RUNTIME=$((END_TIME - START_TIME))

# ===========================================
# Output Generation
# ===========================================
cat > "$OUTPUT_PATH/results.json" <<EOF
{
  "metadata": {
    "script": "$SCRIPT_NAME",
    "version": "$VERSION",
    "timestamp": "$(date -Iseconds)",
    "runtime_seconds": $RUNTIME
  },
  "input": {
    "path": "$INPUT_PATH",
    "files_processed": $METRICS_CALCULATED
  },
  "results": {
    "issues_found": $ISSUES_FOUND,
    "metrics": {
      "key1": "value1",
      "key2": "value2"
    }
  }
}
EOF

echo "Analysis complete: $OUTPUT_PATH/results.json"
echo "Runtime: ${RUNTIME}s"
exit 0

Make it executable:

chmod +x scripts/your-verification-script.sh

Step 6: Build Orchestrator

Copy and adapt: schema-optimization/SKILL.md

Key changes:

1. Update phase names and counts
2. Update input schema (what your workflow needs)
3. Update JSON validation rules (match your phase summaries)
4. Update phase invocation (pass your specific inputs)

Orchestrator checklist:

• Creates timestamp-based session directory
• Spawns Phase 1 with initial inputs
• Validates Phase 1 JSON and report file
• Spawns Phase 2 with Phase 1 report path
• Validates each phase before continuing
• Spawns Phase 4 with script path and output folder
• Validates Phase 4 verification summary
• Aggregates final JSON output
• Handles errors gracefully (no partial runs)

Step 7: Create Sample Outputs

Purpose: Show users what good looks like

What to include:

1. Complete session directory with all phase reports
2. Realistic data (not placeholders)
3. Proper formatting (follow your standards)
4. Phase 4 showing actual verification (script output vs manual)

Tip: Run the workflow once manually, save the outputs, clean them up, move to _samples/

Real-World Examples

Example 1: API Documentation Audit

Goal: Verify API docs match actual code

Phases:

1. Discovery: Find all API endpoints in code
2. Doc Extraction: Extract documented endpoints
3. Gap Analysis: Compare code vs docs
4. Verification: Run script to test each endpoint, verify responses match docs
5. Remediation Plan: Prioritize doc updates

Phase 4 script:

# test_api_endpoints.sh
# Calls each endpoint, compares response schema to documented schema
curl -X GET /api/users | jq . > actual-response.json
jq . docs/api/users.json > documented-response.json
diff actual-response.json documented-response.json

Verification summary:

{
  "endpoints_tested": 42,
  "endpoints_matching_docs": 38,
  "endpoints_diverged": 4,
  "undocumented_endpoints": 2
}

Example 2: Infrastructure Drift Detection

Goal: Verify Terraform state matches actual cloud resources

Phases:

1. State Analysis: Parse Terraform state files
2. Resource Inventory: List what should exist
3. Risk Assessment: Categorize by criticality
4. Verification: Run terraform plan, compare predicted drift vs actual
5. Reconciliation Plan: Steps to fix drift

Phase 4 script:

# verify_terraform_state.sh
terraform plan -detailed-exitcode -out=tfplan
terraform show -json tfplan > actual-plan.json
# Compare manual predictions vs terraform's actual plan

Verification summary:

{
  "resources_checked": 156,
  "drift_predicted": 12,
  "drift_confirmed": 10,
  "drift_false_positives": 2,
  "unexpected_drift": 3
}

Example 3: Dependency Security Audit

Goal: Verify manual security review matches automated scans

Phases:

1. Dependency Tree: Build complete dependency graph
2. Manual Review: Identify suspicious/outdated packages
3. Risk Scoring: Categorize by severity
4. Verification: Run npm audit, safety check, compare findings
5. Remediation: Prioritize updates

Phase 4 script:

# verify_dependencies.sh
npm audit --json > npm-audit-results.json
safety check --json > safety-results.json
# Compare manual findings vs automated scan results

Verification summary:

{
  "packages_scanned": 234,
  "vulnerabilities_predicted": 8,
  "vulnerabilities_confirmed": 7,
  "vulnerabilities_revised": 1,
  "critical_unexpected": 2
}

Common Adaptations

Fewer Phases (3-Phase Workflow)

Phase 1: Initial Analysis
Phase 2: Verification with Tool (critical)
Phase 3: Recommendations

When to use: Simple workflows, quick audits

More Phases (7-Phase Workflow)

Phase 1: Discovery
Phase 2: Collection
Phase 3: Analysis
Phase 4: Deep Dive
Phase 5: Verification 1 (Tool A)
Phase 6: Verification 2 (Tool B)
Phase 7: Synthesis

When to use: Complex workflows, multiple verification steps

Parallel Verification Phases

Phase 4a: Linter Verification
Phase 4b: Test Suite Verification
Phase 4c: Security Scan Verification
Phase 5: Aggregate Verifications

Implementation: Orchestrator spawns 4a/4b/4c concurrently, waits for all, then Phase 5

Checklist: Before You Deploy

Pre-Deployment:

• All phases return valid JSON (test manually)
• All report files are written before JSON return (verify)
• Verification script runs independently (test outside workflow)
• Orchestrator validates all required keys (test with missing keys)
• Session directories are created with timestamps (check naming)
• Reference docs are clear and actionable (have someone else read)
• Sample outputs demonstrate realistic workflow (not placeholders)
• Error handling works (test with bad inputs)

Production Readiness:

• Add logging (orchestrator logs each phase start/end)
• Add monitoring (track success rate, runtime)
• Add alerting (notify on failures)
• Add cleanup (old session directories)
• Document for ops team (how to run, how to debug)
• Load test (can it handle expected volume?)
• Security review (no secrets in logs, proper permissions)

Next Steps

Hands-on practice: Read exercises/exercise-4-build-from-scratch.md and build a 3-phase workflow.

Debugging help: Read GUIDE-03-DEBUGGING-TIPS.md for common issues and solutions.

Deploy to production: Move your workflow out of workspace/lab/ into .claude/skills/ and test with real data.

Next: GUIDE-03-DEBUGGING-TIPS.md