# Cicd Intelligent Recovery

> Loop 3 of the Three-Loop Integrated Development System. CI/CD automation with intelligent failure recovery, root cause analysis, and comprehensive quality validation. Receives implementation from Loop 2, feeds failure patterns back to Loop 1. Achieves 100% test success through automated repair and theater validation. v2.0.0 with explicit agent SOPs. Use when this capability is needed.

- Skill: `tomevault-io/cicd-intelligent-recovery` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/cicd-intelligent-recovery`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/cicd-intelligent-recovery/raw
- Safety review: pending (external: skill-scanner PASS, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/cicd-intelligent-recovery

---


# CI/CD Quality & Debugging Loop (Loop 3)

**Purpose**: Continuous integration with automated failure recovery and authentic quality validation.

**SOP Workflow**: Specification → Research → Planning → Execution → Knowledge

**Output**: 100% test success rate with authentic quality improvements and failure pattern analysis

**Integration**: This is Loop 3 of 3. Receives from `parallel-swarm-implementation` (Loop 2), feeds failure data back to `research-driven-planning` (Loop 1).

**Version**: 2.0.0
**Optimization**: Evidence-based prompting with explicit agent SOPs

---

## When to Use This Skill

Activate this skill when:
- Have complete implementation from Loop 2 (parallel-swarm-implementation)
- Need CI/CD pipeline automation with intelligent recovery
- Require root cause analysis for test failures
- Want automated repair with connascence-aware fixes
- Need validation of authentic quality (no theater)
- Generating failure patterns for Loop 1 feedback

**DO NOT** use this skill for:
- Initial development (use Loop 2 first)
- Manual debugging without CI/CD integration
- Quality checks during development (use Loop 2 theater detection)

---

## Input/Output Contracts

### Input Requirements

```yaml
input:
  loop2_delivery_package:
    location: .claude/.artifacts/loop2-delivery-package.json
    schema:
      implementation: object (complete codebase)
      tests: object (test suite)
      theater_baseline: object (theater metrics from Loop 2)
      integration_points: array[string]
    validation:
      - Must exist and be valid JSON
      - Must include theater_baseline for differential analysis

  ci_cd_failures:
    source: GitHub Actions workflow runs
    format: JSON array of failure objects
    required_fields: [file, line, column, testName, errorMessage, runId]

  github_credentials:
    required: gh CLI authenticated
    check: gh auth status
```

### Output Guarantees

```yaml
output:
  test_success_rate: 100% (guaranteed)

  quality_validation:
    theater_audit: PASSED (no false improvements)
    sandbox_validation: 100% test pass
    differential_analysis: improvement metrics

  failure_patterns:
    location: .claude/.artifacts/loop3-failure-patterns.json
    feeds_to: Loop 1 (next iteration)
    schema:
      patterns: array[failure_pattern]
      recommendations: object (planning/architecture/testing)

  delivery_package:
    location: .claude/.artifacts/loop3-delivery-package.json
    contains:
      - quality metrics (test success, failures fixed)
      - analysis data (root causes, connascence context)
      - validation results (theater, sandbox, differential)
      - feedback for Loop 1
```

---

## Prerequisites

Before starting Loop 3, ensure Loop 2 completion:

```bash
# Verify Loop 2 delivery package exists
test -f .claude/.artifacts/loop2-delivery-package.json && echo "✅ Ready" || echo "❌ Run parallel-swarm-implementation first"

# Load implementation data
npx claude-flow@alpha memory query "loop2_complete" --namespace "integration/loop2-to-loop3"

# Verify GitHub CLI authenticated
gh auth status || gh auth login
```

---

## 8-Step CI/CD Process Overview

```
Step 1: GitHub Hook Integration (Download CI/CD failure reports)
        ↓
Step 2: AI-Powered Analysis (Gemini + 7-agent synthesis with Byzantine consensus)
        ↓
Step 3: Root Cause Detection (Graph analysis + Raft consensus)
        ↓
Step 4: Intelligent Fixes (Program-of-thought: Plan → Execute → Validate → Approve)
        ↓
Step 5: Theater Detection Audit (6-agent Byzantine consensus validation)
        ↓
Step 6: Sandbox Validation (Isolated production-like testing)
        ↓
Step 7: Differential Analysis (Compare to baseline with metrics)
        ↓
Step 8: GitHub Feedback (Automated reporting and loop closure)
```

---

## Step 1: GitHub Hook Integration

**Objective**: Download and process CI/CD pipeline failure reports from GitHub Actions.

**Agent Coordination**: Single orchestrator agent manages data collection.

### Configure GitHub Hooks

```bash
# Install GitHub CLI if needed
which gh || brew install gh

# Authenticate
gh auth login

# Configure webhook listener
gh api repos/{owner}/{repo}/hooks \
  -X POST \
  -f name='web' \
  -f active=true \
  -f events='["check_run", "workflow_run"]' \
  -f config[url]='http://localhost:3000/hooks/github' \
  -f config[content_type]='application/json'
```

### Download Failure Reports

```bash
# Get recent workflow runs
gh run list --repo {owner}/{repo} --limit 10 --json conclusion,databaseId \
  | jq '.[] | select(.conclusion == "failure")' \
  > .claude/.artifacts/failed-runs.json

# Download logs for each failure
cat .claude/.artifacts/failed-runs.json | jq -r '.databaseId' | while read RUN_ID; do
  gh run view $RUN_ID --log \
    > .claude/.artifacts/failure-logs-$RUN_ID.txt
done
```

### Parse Failure Data

```bash
node <<'EOF'
const fs = require('fs');
const failures = [];

// Parse all failure logs
const logFiles = fs.readdirSync('.claude/.artifacts')
  .filter(f => f.startsWith('failure-logs-'));

logFiles.forEach(file => {
  const log = fs.readFileSync(`.claude/.artifacts/${file}`, 'utf8');

  // Extract structured failure data
  const failureMatches = log.matchAll(/FAIL (.+?):(\d+):(\d+)\n(.+?)\n(.+)/g);

  for (const match of failureMatches) {
    failures.push({
      file: match[1],
      line: parseInt(match[2]),
      column: parseInt(match[3]),
      testName: match[4],
      errorMessage: match[5],
      runId: file.match(/failure-logs-(\d+)/)[1]
    });
  }
});

fs.writeFileSync(
  '.claude/.artifacts/parsed-failures.json',
  JSON.stringify(failures, null, 2)
);

console.log(`✅ Parsed ${failures.length} failures`);
EOF
```

**Validation Checkpoint**:
- ✅ Failure data parsed and structured
- ✅ All required fields present (file, line, testName, errorMessage)

---

## Step 2: AI-Powered Analysis

**Objective**: Use Gemini large-context analysis + 7 research agents with Byzantine consensus to examine each failure deeply.

**Evidence-Based Techniques**: Self-consistency, Byzantine consensus, program-of-thought

### Phase 1: Gemini Large-Context Analysis

**Leverage Gemini's 2M token window for full codebase analysis**

```bash
# Analyze failures with full codebase context
/gemini:impact "Analyze CI/CD test failures:

FAILURE DATA:
$(cat .claude/.artifacts/parsed-failures.json)

CODEBASE CONTEXT:
Full repository (all files)

LOOP 2 IMPLEMENTATION:
$(cat .claude/.artifacts/loop2-delivery-package.json)

ANALYSIS OBJECTIVES:
1. Identify cross-file dependencies related to failures
2. Detect failure cascade patterns (root → secondary → tertiary)
3. Analyze what changed between working and failing states
4. Assess system-level architectural impact
5. Identify connascence patterns in failing code

OUTPUT FORMAT:
{
  dependency_graph: { nodes: [files], edges: [dependencies] },
  cascade_map: { root_failures: [], cascaded_failures: [] },
  change_analysis: { changed_files: [], change_impact: [] },
  architectural_impact: { affected_systems: [], coupling_issues: [] }
}"

# Store Gemini analysis
cat .claude/.artifacts/gemini-response.json \
  > .claude/.artifacts/gemini-analysis.json
```

### Phase 2: Parallel Multi-Agent Deep Dive (Self-Consistency)

**7 parallel agents for cross-validation and consensus**

```javascript
// PARALLEL ANALYSIS AGENTS - Evidence-Based Self-Consistency
[Single Message - Spawn All 7 Analysis Agents]:

  // Failure Pattern Research (Dual agents for cross-validation)
  Task("Failure Pattern Researcher 1",
    `Research similar failures in external sources:
    - GitHub issues for libraries we use
    - Stack Overflow questions with similar error messages
    - Documentation of known issues

    Failures to research: $(cat .claude/.artifacts/parsed-failures.json | jq -r '.[].errorMessage')

    For each failure:
    1. Find similar reported issues
    2. Document known solutions with evidence (links, code examples)
    3. Note confidence level (high/medium/low)

    Store findings: .claude/.artifacts/failure-patterns-researcher1.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "research-patterns-1"`,
    "researcher")

  Task("Failure Pattern Researcher 2",
    `Cross-validate findings from Researcher 1:
    - Load: .claude/.artifacts/failure-patterns-researcher1.json
    - Verify each claimed solution independently
    - Check for conflicting solutions
    - Identify most reliable approaches

    For conflicts:
    1. Research both approaches
    2. Determine which is more current/reliable
    3. Flag disagreements for consensus

    Store findings: .claude/.artifacts/failure-patterns-researcher2.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "research-patterns-2"`,
    "researcher")

  // Error Analysis Specialist
  Task("Error Message Analyzer",
    `Deep dive into error messages and stack traces:

    Failures: $(cat .claude/.artifacts/parsed-failures.json)

    For each error message:
    1. Parse error semantics (syntax error vs runtime vs logic)
    2. Extract root cause from stack trace (not just symptoms)
    3. Identify error propagation patterns
    4. Distinguish between:
       - Direct causes (code that threw error)
       - Indirect causes (code that set up failure conditions)

    Apply program-of-thought reasoning:
    "Error X occurred because Y, which was caused by Z"

    Store analysis: .claude/.artifacts/error-analysis.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "error-analysis"`,
    "analyst")

  // Code Context Investigator
  Task("Code Context Investigator",
    `Analyze surrounding code context for failures:

    Load: .claude/.artifacts/parsed-failures.json
    Load: .claude/.artifacts/gemini-analysis.json (for dependency context)

    For each failure:
    1. Read file at failure line ±50 lines
    2. Identify why failure occurs in THIS specific codebase
    3. Find coupling issues (tight coupling → cascading failures)
    4. Analyze code smells that contributed to failure

    Context analysis:
    - Variable/function naming clarity
    - Error handling presence/absence
    - Input validation
    - Edge case handling

    Store findings: .claude/.artifacts/code-context.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "code-context"`,
    "code-analyzer")

  // Test Validity Auditors (Dual agents for critical validation)
  Task("Test Validity Auditor 1",
    `Determine if tests are correctly written:

    Load failures: .claude/.artifacts/parsed-failures.json

    For each failing test:
    1. Is test logic correct? (proper assertions, valid test data)
    2. Is failure indicating real bug or test issue?
    3. Check test quality:
       - Proper setup/teardown
       - Isolated (not depending on other tests)
       - Deterministic (not flaky)

    Categorize:
    - Real bugs: Code is wrong, test is correct
    - Test issues: Code is correct, test is wrong
    - Both wrong: Code and test both have issues

    Store analysis: .claude/.artifacts/test-validity-1.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "test-validity-1"`,
    "tester")

  Task("Test Validity Auditor 2",
    `Cross-validate test analysis from Auditor 1:

    Load: .claude/.artifacts/test-validity-1.json
    Load: .claude/.artifacts/loop2-delivery-package.json (theater baseline)

    Additional checks:
    1. Compare to Loop 2 theater baseline
    2. Check for test theater patterns:
       - Meaningless assertions (expect(1).toBe(1))
       - Over-mocking (mocking the thing being tested)
       - False positives (tests that don't actually test)

    For disagreements with Auditor 1:
    1. Re-examine test thoroughly
    2. Document reasoning for different conclusion
    3. Flag for consensus resolution

    Store analysis: .claude/.artifacts/test-validity-2.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "test-validity-2"`,
    "tester")

  // Dependency Specialist
  Task("Dependency Conflict Detector",
    `Check for dependency-related failures:

    Load failures: .claude/.artifacts/parsed-failures.json

    Analysis steps:
    1. Check package.json/requirements.txt for version conflicts
    2. Identify breaking changes in dependencies:
       - Compare current versions to last working versions
       - Review CHANGELOG files for breaking changes
       - Check deprecation warnings

    3. Analyze transitive dependencies:
       - npm ls or pip list --tree
       - Find version conflicts in dep tree

    4. Check for missing dependencies:
       - ImportError / Cannot find module
       - Missing peer dependencies

    Store findings: .claude/.artifacts/dependency-analysis.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "dependency-analysis"`,
    "analyst")

// Wait for all 7 agents to complete
npx claude-flow@alpha task wait --all --namespace "cicd/analysis"
```

### Phase 3: Synthesis with Byzantine Consensus

**Byzantine fault-tolerant synthesis requires 5/7 agent agreement**

```javascript
[Single Message - Synthesis Coordinator]:
  Task("Analysis Synthesis Coordinator",
    `Synthesize findings from Gemini + 7 agents using Byzantine consensus.

    INPUTS:
    - Gemini Analysis: .claude/.artifacts/gemini-analysis.json
    - Researcher 1: .claude/.artifacts/failure-patterns-researcher1.json
    - Researcher 2: .claude/.artifacts/failure-patterns-researcher2.json
    - Error Analyzer: .claude/.artifacts/error-analysis.json
    - Code Context: .claude/.artifacts/code-context.json
    - Test Auditor 1: .claude/.artifacts/test-validity-1.json
    - Test Auditor 2: .claude/.artifacts/test-validity-2.json
    - Dependency Detector: .claude/.artifacts/dependency-analysis.json

    SYNTHESIS PROCESS:

    1. Cross-Reference Analysis:
       For each failure, collect all agent findings
       Build confidence matrix: which agents agree on root cause

    2. Byzantine Consensus:
       For each root cause claim:
       - Count agent agreement (need 5/7 for consensus)
       - Weight by agent confidence scores
       - Flag conflicts (< 5/7 agreement) for manual review

    3. Consolidate Root Causes:
       - Primary causes: 7/7 agreement (highest confidence)
       - Secondary causes: 5-6/7 agreement (medium confidence)
       - Disputed causes: < 5/7 agreement (flag for review)

    4. Generate Synthesis Report:
       {
         rootCauses: [
           {
             failure: failure_object,
             cause: "root cause description",
             evidence: ["agent1 finding", "agent2 finding"],
             consensus: 7/7 or 6/7 or 5/7,
             confidence: "high" | "medium" | "low"
           }
         ],
         cascadingFailures: [
           { root: failure_id, cascaded: [failure_ids] }
         ],
         quickWins: [ /* easy fixes */ ],
         complexIssues: [ /* require architecture changes */ ]
       }

    VALIDATION:
    - All failures must be categorized
    - Root causes must have >= 5/7 consensus or be flagged
    - Cascading relationships must be validated by Gemini graph

    Store: .claude/.artifacts/analysis-synthesis.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "synthesis-consensus"`,
    "byzantine-coordinator")
```

**Validation Checkpoint**:
- ✅ Gemini analysis complete (dependency graph, cascade map)
- ✅ All 7 agents completed analysis
- ✅ Byzantine consensus achieved (5/7 agreement on root causes)
- ✅ Synthesis report generated with confidence scores

---

## Step 3: Root Cause Detection

**Objective**: Reverse engineer to find cascade issues and true root causes using graph analysis and Raft consensus.

**Evidence-Based Techniques**: Graph algorithms, connascence analysis, Raft consensus

### Phase 1: Parallel Cascade Graph Analysis

**Multiple graph analysts for validation**

```javascript
[Single Message - Parallel Graph Analysis]:

  Task("Failure Graph Analyst 1",
    `Build failure dependency graph using graph algorithms:

    Load: .claude/.artifacts/parsed-failures.json
    Load: .claude/.artifacts/gemini-analysis.json (dependency context)

    GRAPH CONSTRUCTION:
    1. Nodes: Each failure is a node
    2. Edges: Failure A → Failure B if:
       - B's error message references A's file
       - B's file imports A's file
       - B's line number > A's line number in same file
       - Gemini dependency graph shows A → B relationship

    3. Apply graph algorithms:
       - Topological sort to find root nodes (no incoming edges)
       - Calculate cascade depth (max distance from root)
       - Find strongly connected components (circular dependencies)

    4. Identify root causes:
       Root = node with 0 incoming edges OR
       Root = node in cycle with most outgoing edges

    OUTPUT:
    {
      graph: { nodes: [], edges: [] },
      roots: [ /* root failure nodes */ ],
      cascadeMap: { /* depth levels */ },
      circularDeps: [ /* cycles detected */ ]
    }

    Store: .claude/.artifacts/failure-graph-1.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "graph-1"`,
    "analyst")

  Task("Failure Graph Analyst 2",
    `Validate graph structure from Analyst 1:

    Load: .claude/.artifacts/failure-graph-1.json
    Load: .claude/.artifacts/analysis-synthesis.json (consensus data)

    VALIDATION PROCESS:
    1. Cross-check edges:
       - Verify each edge using synthesis consensus
       - Remove edges with low confidence (< 5/7 agreement)
       - Add missing edges identified by consensus

    2. Identify hidden cascades:
       - Indirect cascades (A → B → C, but A → C not obvious)
       - Time-based cascades (A fails first, causes B later)
       - State-based cascades (A leaves bad state, B fails on it)

    3. Validate root cause claims:
       For each claimed root:
       - Verify no hidden dependencies
       - Check if truly primary or just first detected
       - Use 5-Whys: "Why did this fail?" → repeat 5 times

    OUTPUT:
    {
      validatedGraph: { /* corrected graph */ },
      validatedRoots: [ /* confirmed roots */ ],
      hiddenCascades: [ /* newly discovered */ ],
      conflicts: [ /* disagreements with Analyst 1 */ ]
    }

    Store: .claude/.artifacts/failure-graph-2.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "graph-2"`,
    "analyst")

// Wait for graph analysis
npx claude-flow@alpha task wait --namespace "cicd/graph-analysis"
```

### Phase 2: Connascence Analysis

**Identify code coupling that affects fix strategy**

```javascript
[Single Message - Parallel Connascence Detection]:

  Task("Connascence Detector (Name)",
    `Scan for connascence of name: shared variable/function names causing failures.

    Load root causes: .claude/.artifacts/failure-graph-2.json (validatedRoots)

    For each root cause file:
    1. Find all references to symbols (variables, functions, classes)
    2. Identify which files import/use these symbols
    3. Determine if failure requires changes across multiple files

    Connascence of Name = When changing a name requires changing it everywhere

    Impact on fixes:
    - High connascence = Must fix all references atomically
    - Low connascence = Can fix in isolation

    Store: .claude/.artifacts/connascence-name.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "conn-name"`,
    "code-analyzer")

  Task("Connascence Detector (Type)",
    `Scan for connascence of type: type dependencies causing failures.

    Load root causes: .claude/.artifacts/failure-graph-2.json

    For each root cause:
    1. Identify type signatures (function params, return types)
    2. Find all code that depends on these types
    3. Check if failure requires type changes

    Connascence of Type = When changing a type requires updating all users

    TypeScript/Python type hints make this explicit:
    - function foo(x: string) → changing to number affects all callers

    Store: .claude/.artifacts/connascence-type.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "conn-type"`,
    "code-analyzer")

  Task("Connascence Detector (Algorithm)",
    `Scan for connascence of algorithm: shared algorithms causing failures.

    Load root causes: .claude/.artifacts/failure-graph-2.json

    For each root cause:
    1. Identify algorithmic dependencies:
       - Shared validation logic
       - Shared calculation methods
       - Shared state management patterns

    2. Find code using these algorithms
    3. Determine if fix requires algorithm changes across multiple locations

    Connascence of Algorithm = When multiple parts depend on same algorithm

    Example: If authentication algorithm is wrong, must fix:
    - Auth service
    - Token validation
    - Session management

    Store: .claude/.artifacts/connascence-algorithm.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "conn-algorithm"`,
    "code-analyzer")

// Wait for connascence analysis
npx claude-flow@alpha task wait --namespace "cicd/connascence"
```

### Phase 3: Raft Consensus on Root Causes

**Leader-based consensus for final root cause list**

```javascript
[Single Message - Root Cause Consensus]:

  Task("Root Cause Validator",
    `Validate each identified root cause using 5-Whys methodology.

    Load: .claude/.artifacts/failure-graph-2.json (validatedRoots)
    Load: .claude/.artifacts/analysis-synthesis.json (consensus data)

    For each root cause:
    Apply 5-Whys:
    1. Why did this test fail? → [answer]
    2. Why did [answer] happen? → [deeper answer]
    3. Why did [deeper answer] happen? → [deeper still]
    4. Why did [deeper still] happen? → [approaching root]
    5. Why did [approaching root] happen? → TRUE ROOT CAUSE

    Validate:
    - If 5-Whys reveals deeper cause, update root cause
    - If already at true root, confirm
    - Ensure not stopping at symptom

    Store: .claude/.artifacts/root-cause-validation.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "root-validation"`,
    "analyst")

  Task("Root Cause Consensus Coordinator (Raft)",
    `Use Raft consensus to generate final root cause list.

    INPUTS:
    - Graph Analyst 1: .claude/.artifacts/failure-graph-1.json
    - Graph Analyst 2: .claude/.artifacts/failure-graph-2.json
    - Root Cause Validator: .claude/.artifacts/root-cause-validation.json
    - Connascence Detectors: .claude/.artifacts/connascence-*.json

    RAFT CONSENSUS PROCESS:

    1. Leader Election:
       - Graph Analyst 2 is leader (most validated data)
       - Analyst 1 and Validator are followers

    2. Log Replication:
       - Leader proposes root cause list
       - Followers validate against their data
       - Require majority agreement (2/3)

    3. Conflict Resolution:
       For disagreements:
       - Leader's validated graph is authoritative
       - But if Validator's 5-Whys reveals deeper cause, override
       - If Analyst 1 found hidden cascade, add to list

    4. Generate Final Root Cause List:
       {
         roots: [
           {
             failure: failure_object,
             rootCause: "true root cause from 5-Whys",
             cascadedFailures: [failure_ids],
             connascenceContext: {
               name: [affected_files],
               type: [type_dependencies],
               algorithm: [shared_algorithms]
             },
             fixComplexity: "simple" | "moderate" | "complex",
             fixStrategy: "isolated" | "bundled" | "architectural"
           }
         ],
         stats: {
           totalFailures: number,
           rootFailures: number,
           cascadedFailures: number,
           cascadeRatio: percentage
         }
       }

    VALIDATION:
    - All failures accounted for (either root or cascaded)
    - Root causes have 5-Whys validation
    - Connascence context complete for bundled fixes
    - Fix strategies aligned with connascence analysis

    Store: .claude/.artifacts/root-causes-consensus.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "root-consensus"`,
    "raft-manager")
```

**Validation Checkpoint**:
- ✅ Failure dependency graph validated by 2 analysts
- ✅ Connascence analysis complete (name, type, algorithm)
- ✅ Root causes validated with 5-Whys methodology
- ✅ Raft consensus achieved on final root cause list
- ✅ Fix strategies determined based on connascence context

---

## Step 4: Intelligent Fixes

**Objective**: Automated repair with connascence-aware context bundling using program-of-thought structure.

**Evidence-Based Techniques**: Program-of-thought (Plan → Execute → Validate → Approve), self-consistency, consensus approval

### Program-of-Thought Fix Generation

**Explicit Plan → Execute → Validate → Approve for each root cause**

```bash
# Load root causes from Raft consensus
ROOT_CAUSES=$(cat .claude/.artifacts/root-causes-consensus.json | jq -r '.roots[] | @base64')

for ROOT_CAUSE_B64 in $ROOT_CAUSES; do
  ROOT_CAUSE=$(echo "$ROOT_CAUSE_B64" | base64 -d)
  FAILURE_ID=$(echo "$ROOT_CAUSE" | jq -r '.failure.testName')

  echo "=== Fixing Root Cause: $FAILURE_ID ==="

  # PHASE 1: PLANNING
  [Single Message - Fix Strategy Planning]:
    Task("Fix Strategy Planner",
      `MISSION: Plan fix strategy for root cause failure.

      ROOT CAUSE DATA:
      ${ROOT_CAUSE}

      CONNASCENCE CONTEXT:
      Name: $(echo "$ROOT_CAUSE" | jq '.connascenceContext.name')
      Type: $(echo "$ROOT_CAUSE" | jq '.connascenceContext.type')
      Algorithm: $(echo "$ROOT_CAUSE" | jq '.connascenceContext.algorithm')

      PLANNING STEPS (Program-of-Thought):

      Step 1: Understand Root Cause Deeply
      - What is the TRUE root cause (not symptom)?
      - Why did this occur (5-Whys result)?
      - What conditions led to this?

      Step 2: Identify All Affected Files
      - Primary file (where failure occurred)
      - Connascence name files (shared symbols)
      - Connascence type files (type dependencies)
      - Connascence algorithm files (shared logic)

      Step 3: Design Minimal Fix
      - What is the SMALLEST change that fixes root cause?
      - Can we fix in one file or need bundled changes?
      - Are there architectural issues requiring refactor?

      Step 4: Predict Side Effects
      - What else might break from this fix?
      - Are there cascaded failures that will auto-resolve?
      - Are there hidden dependencies not in connascence?

      Step 5: Plan Validation Approach
      - Which tests must pass?
      - Which tests might fail (expected)?
      - Need new tests for edge cases?

      OUTPUT (Detailed Fix Plan):
      {
        rootCause: "description",
        fixStrategy: "isolated" | "bundled" | "architectural",
        files: [
          { path: "file.js", reason: "primary failure location", changes: "description" },
          { path: "file2.js", reason: "connascence of name", changes: "description" }
        ],
        minimalChanges: "description of minimal fix",
        predictedSideEffects: ["effect1", "effect2"],
        validationPlan: {
          mustPass: ["test1", "test2"],
          mightFail: ["test3 (expected)"],
          newTests: ["test4 for edge case"]
        },
        reasoning: "step-by-step explanation of plan"
      }

      Store: .claude/.artifacts/fix-plan-${FAILURE_ID}.json
      Use hooks: npx claude-flow@alpha hooks post-task --task-id "fix-plan-${FAILURE_ID}"`,
      "planner")

  # Wait for planning to complete
  npx claude-flow@alpha task wait --task-id "fix-plan-${FAILURE_ID}"

  # PHASE 2: EXECUTION
  [Single Message - Fix Implementation]:
    Task("Fix Implementation Specialist",
      `MISSION: Execute fix plan with connascence-aware bundled changes.

      LOAD FIX PLAN:
      $(cat .claude/.artifacts/fix-plan-${FAILURE_ID}.json)

      IMPLEMENTATION STEPS (Program-of-Thought):

      Step 1: Load All Affected Files
      - Read each file from fix plan
      - Understand current implementation
      - Locate exact change points

      Step 2: Apply Minimal Fix
      - Implement smallest change from plan
      - Follow fix strategy (isolated vs bundled)
      - For bundled: apply ALL related changes ATOMICALLY

      Step 3: Show Your Work (Reasoning)
      For each change, document:
      - What changed: "Changed X from Y to Z"
      - Why changed: "Because root cause was..."
      - Connascence impact: "Also updated N, T, A files due to connascence"
      - Edge cases handled: "Added validation for..."

      Step 4: Generate Fix Patch
      - Create git diff patch
      - Include all files (atomic bundle)
      - Add descriptive commit message with reasoning

      VALIDATION BEFORE STORING:
      - All files from plan are changed?
      - Changes are minimal (no scope creep)?
      - Connascence context preserved?
      - Code compiles/lints?

      OUTPUT:
      {
        patch: "git diff format",
        filesChanged: ["file1", "file2"],
        changes: [
          { file: "file1", what: "...", why: "...", reasoning: "..." }
        ],
        commitMessage: "descriptive message with reasoning"
      }

      Store: .claude/.artifacts/fix-impl-${FAILURE_ID}.json
      Store patch: .claude/.artifacts/fixes/${FAILURE_ID}.patch
      Use hooks: npx claude-flow@alpha hooks post-edit --memory-key "cicd/fixes/${FAILURE_ID}"`,
      "coder")

  # Wait for implementation
  npx claude-flow@alpha task wait --task-id "fix-impl-${FAILURE_ID}"

  # PHASE 3: VALIDATION (Dual Validators for Self-Consistency)
  [Single Message - Parallel Validation]:
    Task("Fix Validator (Sandbox)",
      `MISSION: Validate fix in isolated sandbox environment.

      LOAD FIX:
      Patch: .claude/.artifacts/fixes/${FAILURE_ID}.patch
      Plan: .claude/.artifacts/fix-plan-${FAILURE_ID}.json

      SANDBOX VALIDATION PROCESS:

      1. Create Isolated Sandbox:
         - Use production-like environment
         - Clean state (no contamination)

      2. Deploy Fix:
         - Apply patch to sandbox codebase
         - Install dependencies
         - Run build/compile

      3. Run Test Suite:
         - Run failing test (must now pass)
         - Run all tests in affected modules
         - Run integration tests

      4. Verify Fix Resolves Root Cause:
         - Check if root cause symptoms gone
         - Verify cascaded failures auto-resolved
         - Ensure no new failures introduced

      5. Check Predicted Side Effects:
         - Did predicted side effects occur?
         - Any unexpected side effects?

      OUTPUT (Validation Report):
      {
        sandboxId: "sandbox-id",
        fixApplied: true/false,
        originalTestPassed: true/false,
        allTestsResult: {
          total: number,
          passed: number,
          failed: number,
          failedTests: [test_names]
        },
        rootCauseResolved: true/false,
        cascadeResolved: true/false,
        newFailures: [failures],
        sideEffects: {
          predicted: [matched_predictions],
          unexpected: [surprises]
        },
        verdict: "PASS" | "FAIL",
        reasoning: "detailed explanation"
      }

      Store: .claude/.artifacts/fix-validation-sandbox-${FAILURE_ID}.json
      Use hooks: npx claude-flow@alpha hooks post-task --task-id "fix-val-sandbox-${FAILURE_ID}"`,
      "tester")

    Task("Fix Validator (Theater)",
      `MISSION: Audit fix for theater - ensure authentic improvement, not symptom masking.

      LOAD FIX:
      Patch: .claude/.artifacts/fixes/${FAILURE_ID}.patch
      Theater Baseline: .claude/.artifacts/loop2-delivery-package.json (theater_baseline)

      THEATER DETECTION PROCESS:

      1. Fix Theater Scan:
         - Did fix comment out failing test? ❌ THEATER
         - Did fix add "return true" without logic? ❌ THEATER
         - Did fix suppress error without handling? ❌ THEATER

      2. Mock Escalation Check:
         - Did fix add more mocks instead of fixing code? ❌ THEATER
         - Example: jest.mock('./auth', () => ({ login: () => true }))
         - This masks failure, doesn't fix it

      3. Coverage Theater Check:
         - Did fix add meaningless tests for coverage? ❌ THEATER
         - Example: test('filler', () => expect(1).toBe(1))

      4. Compare to Loop 2 Baseline:
         - Is theater level same or reduced?
         - Any new theater introduced?
         - Calculate theater delta

      5. Authentic Improvement Validation:
         - Does fix address root cause genuinely?
         - Is improvement real or illusory?
         - Will fix hold up in production?

      OUTPUT (Theater Report):
      {
        theaterScan: {
          fixTheater: true/false,
          mockEscalation: true/false,
          coverageTheater: true/false,
          details: [specific_instances]
        },
        baselineComparison: {
          loop2Theater: number,
          currentTheater: number,
          delta: number (negative = improvement)
        },
        authenticImprovement: true/false,
        verdict: "PASS" | "FAIL",
        reasoning: "detailed explanation"
      }

      Store: .claude/.artifacts/fix-validation-theater-${FAILURE_ID}.json
      Use hooks: npx claude-flow@alpha hooks post-task --task-id "fix-val-theater-${FAILURE_ID}"`,
      "theater-detection-audit")

  # Wait for both validators
  npx claude-flow@alpha task wait --namespace "cicd/validation-${FAILURE_ID}"

  # PHASE 4: CONSENSUS APPROVAL
  [Single Message - Fix Approval Decision]:
    Task("Fix Approval Coordinator",
      `MISSION: Review fix and validations, make consensus-based approval decision.

      INPUTS:
      - Fix Plan: .claude/.artifacts/fix-plan-${FAILURE_ID}.json
      - Fix Implementation: .claude/.artifacts/fix-impl-${FAILURE_ID}.json
      - Sandbox Validation: .claude/.artifacts/fix-validation-sandbox-${FAILURE_ID}.json
      - Theater Validation: .claude/.artifacts/fix-validation-theater-${FAILURE_ID}.json

      APPROVAL CRITERIA (ALL must pass):

      1. Sandbox Validation: PASS
         - Original test passed: true
         - Root cause resolved: true
         - No new failures: true OR predicted failures only
         - Verdict: PASS

      2. Theater Validation: PASS
         - No new theater introduced: true
         - Authentic improvement: true
         - Theater delta: <= 0 (same or reduced)
         - Verdict: PASS

      3. Implementation Quality:
         - Changes match plan: true
         - Minimal fix applied: true
         - Connascence respected: true

      DECISION LOGIC:

      IF both validators PASS:
        APPROVE → Apply fix to codebase

      IF sandbox PASS but theater FAIL:
        REJECT → Fix masks problem, not genuine
        Feedback: "Fix introduces theater: [details]"
        Action: Regenerate fix without theater

      IF sandbox FAIL:
        REJECT → Fix doesn't work or breaks other tests
        Feedback: "Sandbox validation failed: [details]"
        Action: Revise fix plan, consider architectural fix

      OUTPUT (Approval Decision):
      {
        decision: "APPROVED" | "REJECTED",
        reasoning: "detailed explanation",
        validations: {
          sandbox: "PASS/FAIL",
          theater: "PASS/FAIL"
        },
        action: "apply_fix" | "regenerate_without_theater" | "revise_plan",
        feedback: "feedback for retry if rejected"
      }

      IF APPROVED:
        git apply .claude/.artifacts/fixes/${FAILURE_ID}.patch
        echo "✅ Fix applied: ${FAILURE_ID}"
      ELSE:
        echo "❌ Fix rejected: ${FAILURE_ID}"
        echo "Feedback: $(cat .claude/.artifacts/fix-approval-${FAILURE_ID}.json | jq -r '.feedback')"

      Store: .claude/.artifacts/fix-approval-${FAILURE_ID}.json
      Use hooks: npx claude-flow@alpha hooks post-task --task-id "fix-approval-${FAILURE_ID}"`,
      "hierarchical-coordinator")
done

# Generate fix summary
node <<'EOF'
const fs = require('fs');
const approvals = fs.readdirSync('.claude/.artifacts')
  .filter(f => f.startsWith('fix-approval-'))
  .map(f => JSON.parse(fs.readFileSync(`.claude/.artifacts/${f}`, 'utf8')));

const summary = {
  total: approvals.length,
  approved: approvals.filter(a => a.decision === 'APPROVED').length,
  rejected: approvals.filter(a => a.decision === 'REJECTED').length,
  approvalRate: (approvals.filter(a => a.decision === 'APPROVED').length / approvals.length * 100).toFixed(1)
};

console.log(`✅ Fix Summary: ${summary.approved}/${summary.total} approved (${summary.approvalRate}%)`);

fs.writeFileSync(
  '.claude/.artifacts/fix-summary.json',
  JSON.stringify(summary, null, 2)
);
EOF
```

**Validation Checkpoint**:
- ✅ All root causes have fix plans (program-of-thought planning)
- ✅ Fixes implemented with connascence-aware bundling
- ✅ Dual validation (sandbox + theater) complete
- ✅ Consensus approval for each fix
- ✅ Approved fixes applied to codebase

---

## Step 5: Theater Detection Audit

**Objective**: Validate that fixes are authentic improvements using 6-agent Byzantine consensus.

**Evidence-Based Techniques**: Self-consistency (multiple theater detectors), Byzantine consensus

### Load Loop 2 Theater Baseline

```bash
# Get theater baseline from Loop 2
BASELINE=$(npx claude-flow@alpha memory query "loop2_theater_baseline" \
  --namespace "integration/loop3-validation")

echo "$BASELINE" > .claude/.artifacts/loop2-theater-baseline.json
```

### 6-Agent Theater Detection with Byzantine Consensus

```javascript
[Single Message - Parallel Theater Detection]:

  // Theater Detection Specialists (Multiple Perspectives)
  Task("Theater Detector (Code)",
    `Scan for completion theater in fixed code:

    COMPLETION THEATER PATTERNS:
    - TODOs marked done but not implemented
    - Empty functions returning success (return true;)
    - Mock implementations in production code
    - Commented-out functionality with "FIXED" comments
    - Placeholder implementations (throw new Error('Not implemented'))

    Scan all fixed files: $(ls .claude/.artifacts/fixes/*.patch | xargs -I {} basename {})

    For each pattern found:
    1. File location
    2. Pattern type
    3. Severity (critical/medium/low)
    4. Evidence (code snippet)

    Store: .claude/.artifacts/theater-code.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "theater-code"`,
    "theater-detection-audit")

  Task("Theater Detector (Tests)",
    `Scan for test theater in test files:

    TEST THEATER PATTERNS:
    - Meaningless assertions (expect(1).toBe(1))
    - Tests that don't test anything (empty test bodies)
    - 100% mocked tests with no integration validation
    - Tests that only test mocks, not real code
    - Disabled tests without explanation (test.skip)
    - Over-broad assertions (expect(result).toBeTruthy())

    Scan test files modified in fixes

    For each pattern:
    1. Test name
    2. Pattern type
    3. Why it's theater
    4. Evidence (test code)

    Store: .claude/.artifacts/theater-tests.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "theater-tests"`,
    "tester")

  Task("Theater Detector (Docs)",
    `Scan for documentation theater:

    DOC THEATER PATTERNS:
    - Docs that don't match code (incorrect)
    - Copied templates without customization
    - Placeholder text (Lorem ipsum, TODO, TBD)
    - Documentation claiming features not implemented
    - Outdated examples that don't work

    Scan documentation files modified in fixes

    For each pattern:
    1. Doc file
    2. Pattern type
    3. Why it's theater
    4. Evidence (doc snippet vs code reality)

    Store: .claude/.artifacts/theater-docs.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "theater-docs"`,
    "docs-writer")

  // Reality Validation Agents
  Task("Sandbox Execution Validator",
    `Execute code in isolated sandbox to verify it actually runs.

    REALITY VALIDATION:
    1. Create fresh sandbox
    2. Deploy all fixes
    3. Run with realistic inputs (not trivial examples)
    4. Test edge cases, error cases, invalid inputs
    5. Verify outputs are co

…(truncated)