2674 Task 25 Fix Multi Claude Reliability 5a6b6484

TASK-25: Fix Multi-Claude Workflow Reliability (30% → 90%)

Created: 2025-11-01 Assignee: Multi-Claude Workflow (dogfooding) Priority: High Status: Planning Target: v4.5.0

Problem Statement

Current state: Multi-Claude workflows succeed 30% of the time (3/10 test workflows)

Main failure mode: Marker timeout in Phase 3 (Testing)

[15:24:49] Waiting for file: .agent/tasks/poc-1762006598-tests-done
[15:26:51] ❌ Timeout waiting for file
[15:26:51] ❌ Testing phase timeout - no completion marker

Root causes:

1. Sub-Claude instances don't always invoke marker skill
2. No retry logic when marker creation fails
3. No self-monitoring in sub-Claude (doesn't detect own timeout)
4. Orchestrator can't distinguish "working" from "stuck"
5. No recovery mechanism to resume from last successful phase

Impact: Users can't rely on multi-Claude workflows for production use

Success Criteria

Target: 90% success rate (9/10 workflows complete successfully)

Metrics:

• Marker creation reliability: 95%+ (was ~60%)
• Phase transition success: 95%+ (was ~70%)
• Automatic recovery from timeouts: 80%+
• User intervention needed: <10% (was ~70%)

Test plan:

• 10 simple POC workflows (1-2 file changes)
• 10 medium workflows (3-5 file changes)
• 5 complex workflows (6+ files, integration tests)
• All run without manual intervention

Solution Design

1. Automatic Retry Logic

Problem: Marker skill invoked but creation fails silently

Solution: Orchestrator detects missing marker and retries phase

Implementation (navigator-multi-claude.sh):

wait_for_marker_with_retry() {
  local marker_file=$1
  local max_retries=${2:-1}  # Default: retry once
  local timeout=${3:-120}    # Default: 2 minutes

  for attempt in $(seq 1 $((max_retries + 1))); do
    echo "[$(date +%H:%M:%S)] Attempt $attempt: Waiting for $marker_file"

    if wait_for_file "$marker_file" "$timeout"; then
      echo "[$(date +%H:%M:%S)] ✅ Marker found: $marker_file"
      return 0
    fi

    if [ $attempt -le $max_retries ]; then
      echo "[$(date +%H:%M:%S)] ⚠️  Retry: Marker not found, restarting phase..."
      restart_current_phase
    else
      echo "[$(date +%H:%M:%S)] ❌ Failed after $attempt attempts"
      return 1
    fi
  done
}

Files modified:

• scripts/navigator-multi-claude.sh (retry wrapper)
• scripts/navigator-multi-claude-poc.sh (POC retry)

Testing:

• Simulate marker failure (delete marker mid-creation)
• Verify orchestrator retries
• Confirm success on retry

2. Timeout Detection in Sub-Claude

Problem: Sub-Claude doesn't know it's stuck, orchestrator times out externally

Solution: Sub-Claude monitors own progress, exits if stuck

Implementation (new file: scripts/sub-claude-monitor.sh):

#!/bin/bash
# Sub-Claude self-monitoring wrapper
# Runs alongside headless Claude, kills if stuck

PHASE=$1
TIMEOUT=${2:-180}  # 3 minutes default
PID=$3

start_time=$(date +%s)

while true; do
  sleep 10

  current_time=$(date +%s)
  elapsed=$((current_time - start_time))

  # Check if process still alive
  if ! kill -0 $PID 2>/dev/null; then
    echo "[Monitor] Process $PID completed"
    exit 0
  fi

  # Check if timeout exceeded
  if [ $elapsed -gt $TIMEOUT ]; then
    echo "[Monitor] ⚠️  Timeout exceeded ($elapsed > $TIMEOUT)"
    echo "[Monitor] Checking for progress markers..."

    # Check if marker exists (success despite timeout)
    if [ -f ".agent/tasks/$SESSION_ID-$PHASE-done" ]; then
      echo "[Monitor] ✅ Marker found, phase completed"
      exit 0
    fi

    # No marker, kill stuck process
    echo "[Monitor] ❌ No marker, killing stuck process"
    kill -9 $PID
    exit 1
  fi
done

Usage in orchestrator:

# Start Claude in background
claude -p "$PROMPT" --resume "$SESSION_ID" &
CLAUDE_PID=$!

# Start monitor
./scripts/sub-claude-monitor.sh "$PHASE" "$TIMEOUT" "$CLAUDE_PID" &
MONITOR_PID=$!

# Wait for Claude to finish
wait $CLAUDE_PID
CLAUDE_EXIT=$?

# Kill monitor
kill $MONITOR_PID 2>/dev/null

# Check exit code
if [ $CLAUDE_EXIT -ne 0 ]; then
  echo "❌ Sub-Claude exited with error: $CLAUDE_EXIT"
fi

Files:

• scripts/sub-claude-monitor.sh (new)
• scripts/navigator-multi-claude.sh (integrate monitor)

Testing:

• Simulate stuck Claude (infinite loop)
• Verify monitor kills after timeout
• Confirm orchestrator handles exit code

3. Phase Recovery Mechanism

Problem: Workflow fails completely if one phase times out

Solution: Save phase state, allow resume from last successful phase

Implementation (state file: .agent/tasks/SESSION_ID-state.json):

{
  "session_id": "poc-1762006598",
  "task": "Add hello world function",
  "phases_completed": ["phase0", "phase1", "phase2"],
  "current_phase": "phase3",
  "phase_attempts": {
    "phase3": 2
  },
  "started_at": "2025-11-01T15:24:49Z",
  "last_update": "2025-11-01T15:26:51Z"
}

Resume logic:

resume_workflow() {
  local session_id=$1
  local state_file=".agent/tasks/$session_id-state.json"

  if [ ! -f "$state_file" ]; then
    echo "❌ No state file found: $state_file"
    return 1
  fi

  # Parse state
  local completed=$(jq -r '.phases_completed | join(",")' "$state_file")
  local current=$(jq -r '.current_phase' "$state_file")

  echo "📋 Resuming workflow: $session_id"
  echo "   Completed phases: $completed"
  echo "   Current phase: $current"
  echo ""
  echo "Resume from current phase? [Y/n]"

  read -r response
  if [[ "$response" =~ ^([yY][eE][sS]|[yY]|)$ ]]; then
    run_phase "$current"
  fi
}

Files:

• scripts/navigator-multi-claude.sh (state tracking)
• scripts/resume-workflow.sh (new, standalone resume tool)

Testing:

• Start workflow, kill during Phase 3
• Run resume script
• Verify Phase 3 restarts from checkpoint

4. Marker Verification & Logging

Problem: Marker creation fails silently, no visibility into why

Solution: Enhanced logging and verification in marker skill

Implementation (skills/nav-marker/SKILL.md):

# After marker creation
if [ -f "$MARKER_FILE" ]; then
  # Verify file is readable and non-empty
  if [ -s "$MARKER_FILE" ]; then
    # Log success with metadata
    echo "[$(date +%H:%M:%S)] ✅ Marker created: $MARKER_FILE"
    echo "[$(date +%H:%M:%S)]    Size: $(wc -c < "$MARKER_FILE") bytes"
    echo "[$(date +%H:%M:%S)]    Phase: $PHASE"

    # Create verification checksum
    md5sum "$MARKER_FILE" > "${MARKER_FILE}.md5"

    # Log to central marker log
    echo "$(date -Iseconds) | $PHASE | SUCCESS | $MARKER_FILE" >> .agent/.marker-log
  else
    echo "[$(date +%H:%M:%S)] ⚠️  Marker empty: $MARKER_FILE"
    echo "$(date -Iseconds) | $PHASE | EMPTY | $MARKER_FILE" >> .agent/.marker-log
  fi
else
  echo "[$(date +%H:%M:%S)] ❌ Marker creation failed: $MARKER_FILE"
  echo "$(date -Iseconds) | $PHASE | FAILED | $MARKER_FILE" >> .agent/.marker-log
fi

Orchestrator verification:

verify_marker() {
  local marker_file=$1

  # Check exists
  if [ ! -f "$marker_file" ]; then
    echo "❌ Marker missing: $marker_file"
    return 1
  fi

  # Check non-empty
  if [ ! -s "$marker_file" ]; then
    echo "❌ Marker empty: $marker_file"
    return 1
  fi

  # Verify checksum if available
  if [ -f "${marker_file}.md5" ]; then
    if md5sum -c "${marker_file}.md5" >/dev/null 2>&1; then
      echo "✅ Marker verified: $marker_file"
      return 0
    else
      echo "⚠️  Marker checksum mismatch: $marker_file"
      return 1
    fi
  fi

  echo "✅ Marker exists: $marker_file"
  return 0
}

Files:

• skills/nav-marker/SKILL.md (enhanced logging)
• scripts/navigator-multi-claude.sh (verification)

Testing:

• Create markers in all phases
• Verify logs capture all events
• Simulate corrupted marker, verify detection

5. Improved Sub-Claude Prompts

Problem: Sub-Claude doesn't consistently invoke marker skill

Solution: Explicit marker invocation in phase prompts

Current prompt (vague):

Complete Phase 3: Testing

1. Run tests
2. Verify results
3. Document findings
4. Signal completion

Improved prompt (explicit):

Complete Phase 3: Testing

CRITICAL: You MUST create a completion marker when done.

Steps:
1. Run tests with pytest
2. Verify all tests pass
3. Document test results
4. Create completion marker:

   INVOKE: nav-marker skill with name "phase3-testing-complete"

   This marker is REQUIRED for the orchestrator to proceed.
   Do NOT skip this step.

Session ID: {SESSION_ID}
Marker file: .agent/tasks/{SESSION_ID}-tests-done

Files:

• scripts/navigator-multi-claude.sh (update all phase prompts)
• scripts/navigator-multi-claude-poc.sh (update POC prompts)

Testing:

• Run workflow with updated prompts
• Monitor marker creation rate
• Target: 95%+ marker creation

Implementation Plan

Phase 1: Foundation (Days 1-2)

1. Create sub-claude-monitor.sh
2. Implement state tracking (.agent/tasks/SESSION_ID-state.json)
3. Add marker verification logic
4. Update marker skill logging

Deliverable: Infrastructure ready for retry/recovery

Phase 2: Retry Logic (Days 3-4)

1. Implement wait_for_marker_with_retry()
2. Add restart_current_phase()
3. Update all phase transitions
4. Test retry on simulated failures

Deliverable: Orchestrator can retry failed phases

Phase 3: Recovery (Days 5-6)

1. Create resume-workflow.sh script
2. Implement state persistence
3. Add resume option to orchestrator
4. Test mid-workflow interruption recovery

Deliverable: Users can resume interrupted workflows

Phase 4: Prompts & Verification (Day 7)

1. Update all sub-Claude phase prompts
2. Add explicit marker invocation instructions
3. Enhance marker verification
4. Create central marker log

Deliverable: Sub-Claudes reliably create markers

Phase 5: Testing (Days 8-10)

1. Run 10 simple POC workflows
2. Run 10 medium workflows
3. Run 5 complex workflows
4. Measure success rate
5. Debug remaining failures

Deliverable: 90%+ success rate verified

Testing Strategy

Unit Tests (Per Component)

• test-retry-logic.sh - Simulates marker failures
• test-monitor.sh - Simulates stuck Claude
• test-recovery.sh - Simulates mid-workflow kill
• test-verification.sh - Tests marker validation

Integration Tests (Full Workflows)

• Simple (1-2 files): "Add hello world function"
• Medium (3-5 files): "Implement JWT authentication"
• Complex (6+ files): "Add OAuth2 with Google provider"

Success Metrics

• Marker creation: 95%+ (measure via .agent/.marker-log)
• Phase transitions: 95%+ (measure via state.json)
• Recovery success: 80%+ (measure via resume tests)
• Overall success: 90%+ (9/10 workflows complete)

Rollout Plan

v4.5.0-alpha (Internal Testing)

• Ship to Navigator maintainer only
• Run 25 test workflows
• Collect failure logs
• Iterate on fixes

v4.5.0-beta (Early Adopters)

• Ship as pre-release
• Announce in release notes
• Ask for bug reports
• Target: 85%+ success rate

v4.5.0 (Stable)

• Ship as stable release
• Update status: Experimental → Beta
• Document known limitations
• Provide troubleshooting guide

Files to Modify

New Files

• scripts/sub-claude-monitor.sh (~60 lines)
• scripts/resume-workflow.sh (~100 lines)
• tests/test-retry-logic.sh (~50 lines)
• tests/test-monitor.sh (~40 lines)
• tests/test-recovery.sh (~60 lines)
• .agent/.marker-log (log file)

Modified Files

• scripts/navigator-multi-claude.sh (+200 lines)
  • Retry logic
  • State tracking
  • Monitor integration
  • Verification
• scripts/navigator-multi-claude-poc.sh (+100 lines)
  • Retry logic
  • Updated prompts
• skills/nav-marker/SKILL.md (+30 lines)
  • Enhanced logging
  • Verification
• skills/nav-start/SKILL.md (+10 lines)
  • Update status: Experimental → Beta

Documentation

• RELEASE-NOTES-v4.5.0.md (new)
• .agent/sops/development/multi-claude-troubleshooting.md (new)
• scripts/POC-LEARNINGS.md (update with v4.5 improvements)

Total estimated: ~600 new lines, ~340 modified lines

Risk Assessment

High Risk

• Monitor killing legitimate Claude instances

  • Mitigation: Conservative timeouts (3+ minutes)
  • Mitigation: Progress marker check before kill

• Retry logic infinite loops

  • Mitigation: Max 1 retry per phase
  • Mitigation: Global workflow timeout (30 minutes)

Medium Risk

• State file corruption

  • Mitigation: JSON validation before parse
  • Mitigation: Atomic writes with temp files

• Marker checksum false positives

  • Mitigation: Make checksum optional
  • Mitigation: Fallback to file existence check

Low Risk

• Logging fills disk
  • Mitigation: Log rotation (keep last 100 entries)
  • Mitigation: Cleanup on workflow completion

Success Indicators

Week 1: Infrastructure complete, retries working Week 2: 70%+ success rate (improvement from 30%) Week 3: 85%+ success rate (beta quality) Week 4: 90%+ success rate (stable quality)

Ship when: 90%+ success rate achieved consistently

Execution Method

Dogfooding: Use multi-Claude workflow to implement multi-Claude fixes

Command:

navigator-multi-claude.sh "Implement v4.5.0 multi-Claude reliability fixes from TASK-25"

Expected behavior:

• Phase 1: Planning (read this task doc, create implementation plan)
• Phase 2: Implementation (modify scripts, add retry logic, etc.)
• Phase 3-4: Testing + Docs (run tests, update release notes)
• Phase 5: Review (verify all changes, check for regressions)

If it fails: Proves the problem, implement fixes manually If it succeeds: Validates the approach, release v4.5.0

Next Actions

1. Commit this task doc
2. Run multi-Claude workflow with this task
3. Monitor for marker timeouts
4. Iterate on fixes based on real failure modes

Ready to execute? 🚀