tools-only Multi-Claude Workflow Troubleshooting
Category: Development Created: 2025-11-02 Last Updated: 2025-11-02 Applies to: Navigator v4.5.0+
Context
When to use this SOP: When multi-Claude workflows fail, timeout, or produce unexpected results. This guide helps diagnose and recover from common failure modes.
Problem it solves: Multi-Claude workflows involve multiple headless Claude instances coordinating via markers. Failures can occur at phase boundaries, during marker creation, or due to timeouts. This SOP provides systematic diagnosis and recovery procedures.
Prerequisites:
- Navigator v4.5.0+ installed
- Basic understanding of multi-Claude workflow phases
- Terminal access to project directory
Common Failure Modes
1. Marker Timeout (Most Common)
Symptoms:
[15:24:49] Waiting for file: .agent/tasks/poc-XXX-done
[15:26:51] ❌ Timeout waiting for file
[15:26:51] ❌ Implementation phase timeout - no completion marker
Root causes:
- Sub-Claude didn't invoke marker skill
- Marker skill invoked but creation failed
- Claude process hung/crashed
- File system permissions issue
Diagnosis steps:
# 1. Check if marker file exists (might be created after timeout)
ls -la .agent/tasks/*-done
# 2. Check marker log for events
tail -50 .agent/.marker-log
# 3. Check if Claude process still running
ps aux | grep claude
# 4. Check state file
cat .agent/tasks/{session-id}-state.json | jq .
Recovery:
# Option 1: Resume workflow (if state file exists)
./scripts/resume-workflow.sh {session-id}
# Option 2: Manual marker creation (if work completed)
touch .agent/tasks/{session-id}-{phase}-done
# Option 3: Restart from scratch
./scripts/navigator-multi-claude-poc.sh "Your task"
2. Phase Retry Loop
Symptoms:
[15:27:00] ⚠️ Retry attempt 1 of 2 for phase: implementation
[15:29:05] ⚠️ Retry attempt 2 of 2 for phase: implementation
[15:31:10] ❌ Phase failed after 2 attempts: implementation
Root cause: Phase consistently failing marker creation (retry logic activated but unsuccessful)
Diagnosis:
# 1. Check marker log for pattern
grep "implementation" .agent/.marker-log
# 2. Check if files were actually modified
git status --short
# 3. Review state file attempts
cat .agent/tasks/{session-id}-state.json | jq '.phase_attempts'
Recovery:
# If work was completed (files modified):
touch .agent/tasks/{session-id}-done
./scripts/resume-workflow.sh {session-id}
# If work not started:
# Fix underlying issue, then restart phase manually
3. Monitor False Positive (Process Killed Prematurely)
Symptoms:
[Monitor] ⚠️ Timeout exceeded (180 > 180)
[Monitor] ❌ No marker, killing stuck process
# But work was actually in progress
Root cause: Phase took longer than timeout (180s default)
Diagnosis:
# 1. Check monitor log
grep "Monitor" .agent/.marker-log
# 2. Check if partial work exists
git diff
# 3. Verify process was killed
ps aux | grep claude
Recovery:
# 1. Increase timeout for complex phases
# Edit navigator-multi-claude.sh, increase timeout:
# wait_for_marker_with_retry "$file" "phase" 1 300 # 5 min instead of 3
# 2. Resume workflow
./scripts/resume-workflow.sh {session-id}
4. State File Corruption
Symptoms:
cat: .agent/tasks/SESSION-state.json: invalid JSON
Root cause: Incomplete write or concurrent modification
Diagnosis:
# Try to parse JSON
jq empty .agent/tasks/{session-id}-state.json
# If invalid, check for backup
ls -la .agent/tasks/{session-id}-state.json*
Recovery:
# Option 1: Restore from backup (if exists)
cp .agent/tasks/{session-id}-state.json.bak .agent/tasks/{session-id}-state.json
# Option 2: Manually reconstruct state
cat > .agent/tasks/{session-id}-state.json <<EOF
{
"session_id": "{session-id}",
"task": "{TASK-XX}",
"phases_completed": ["phase0", "phase1"],
"current_phase": "phase2",
"status": "failed",
"last_update": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF
5. Empty Marker Files
Symptoms:
[15:30:00] ⚠️ Marker exists but is invalid (empty or corrupted)
Root cause: Marker created but not written to
Diagnosis:
# Check marker file size
ls -lh .agent/tasks/{session-id}-done
# Check if empty
[ -s .agent/tasks/{session-id}-done ] && echo "Has content" || echo "Empty"
Recovery:
# Markers can be empty - this is usually OK
# If workflow complains, recreate with content:
echo "completed" > .agent/tasks/{session-id}-done
6. Parallel Phase Deadlock
Symptoms:
[15:30:00] Waiting for parallel phases to complete...
# Hangs indefinitely
Root cause: One parallel phase (testing/docs) completed, other stuck
Diagnosis:
# Check which markers exist
ls -la .agent/tasks/{session-id}-tests-done
ls -la .agent/tasks/{session-id}-docs-done
# Check if processes running
ps aux | grep "claude -p"
Recovery:
# Kill hung processes
pkill -f "claude -p"
# Create missing marker manually if work done
touch .agent/tasks/{session-id}-tests-done
# Resume
./scripts/resume-workflow.sh {session-id}
Diagnostic Checklist
When a workflow fails, run through this checklist:
# 1. What phase failed?
tail -20 {workflow-output} # Check last logs
# 2. Was marker created?
ls -la .agent/tasks/*{session-id}*done
# 3. What does marker log show?
tail -50 .agent/.marker-log
# 4. What's the workflow state?
cat .agent/tasks/{session-id}-state.json | jq .
# 5. Was work actually completed?
git status --short
# 6. Are Claude processes still running?
ps aux | grep claude
# 7. What files were created/modified?
git diff --stat
Recovery Decision Tree
Workflow failed?
├─ Marker exists?
│ ├─ YES: Work completed, create marker manually → Resume
│ └─ NO: Work incomplete → Check state file
│
├─ State file exists?
│ ├─ YES: Resume from last phase
│ └─ NO: Restart workflow from scratch
│
├─ Work was done (git diff shows changes)?
│ ├─ YES: Create marker, resume
│ └─ NO: Retry phase or restart
│
└─ Timeout too short?
├─ YES: Increase timeout, retry
└─ NO: Investigate root cause
Preventive Measures
1. Monitor Marker Logs During Workflow
# In separate terminal, watch marker creation in real-time
tail -f .agent/.marker-log
2. Use Longer Timeouts for Complex Tasks
# For workflows likely to take >3 minutes per phase
# Edit scripts/navigator-multi-claude.sh
# Change: wait_for_marker_with_retry "$file" "phase" 1 180
# To: wait_for_marker_with_retry "$file" "phase" 1 300
3. Verify Before Running
# Ensure clean state
git status --short # Should be clean or have expected changes
# Check no orphaned processes
ps aux | grep claude | grep -v grep
# Verify marker log accessible
touch .agent/.marker-log && ls -la .agent/.marker-log
Advanced Debugging
Enable Verbose Logging
# Add debug output to orchestrator
export DEBUG=1
./scripts/navigator-multi-claude-poc.sh "Your task"
# Check Claude Code logs
tail -f ~/.claude/logs/claude.log
Inspect Sub-Claude Prompts
# View exactly what sub-Claude received
# Look in navigator-multi-claude.sh for prompt construction
grep -A 20 "claude -p" scripts/navigator-multi-claude.sh
Test Monitor Separately
# Simulate Claude process
sleep 60 &
PID=$!
# Test monitor
./scripts/sub-claude-monitor.sh "test-phase" 30 $PID /tmp/test-marker
# Should timeout and kill process after 30s
Real Examples
Example 1: TASK-25 Implementation Failure
Scenario: Multi-Claude implementing TASK-25 failed at Phase 2
Diagnosis:
$ ls -la .agent/tasks/poc-1762086614*
-rw-r--r-- plan.md
# No -done marker found
$ git status --short
M scripts/sub-claude-monitor.sh # Work WAS done
M scripts/resume-workflow.sh
?? tests/
$ tail .agent/.marker-log
[2025-11-02T12:35:00Z] ❌ implementation: Timeout (no marker after 180s)
Conclusion: Implementation completed but marker not created
Fix:
# Work done, create marker manually
touch .agent/tasks/poc-1762086614-done
# Resume workflow
./scripts/resume-workflow.sh poc-1762086614
Example 2: Review Phase Success
Scenario: Manual review phase invocation worked
What worked:
$ claude -p "Review task..." &
PID=$!
# Monitor showed progress
$ tail -f .agent/.marker-log
[2025-11-02T12:50:00Z] ✅ review: marker created
# Marker file created
$ ls -la .agent/tasks/poc-1762086614-review-done
-rw-r--r-- 0 bytes # Empty is OK
Lesson: Explicit marker instructions in prompts increased success rate
When to Give Up and Restart
Restart from scratch if:
- State file irreparably corrupted
- Multiple phases failed consecutively
- Workflow state doesn't match reality (git diff vs state.json)
- Session ID conflicts (multiple workflows with same ID)
How to clean restart:
# 1. Kill all processes
pkill -f "navigator-multi-claude"
# 2. Clean state files
rm -f .agent/tasks/*-state.json
# 3. Clean marker files (optional, depends on situation)
rm -f .agent/tasks/*-done
# 4. Commit or stash changes
git add . && git stash # Or commit if good
# 5. Start fresh
./scripts/navigator-multi-claude-poc.sh "Your task"
Success Indicators
Workflow is healthy when:
- ✅ Each phase creates marker within timeout
- ✅ Marker log shows steady progress
- ✅ State file updates after each phase
- ✅ Git diff shows expected changes
- ✅ No orphaned Claude processes
Monitor health with:
# Success rate over last 10 workflows
grep "✅" .agent/.marker-log | wc -l
grep "❌" .agent/.marker-log | wc -l
# Average phase duration
# (requires timestamp parsing, TBD)
Related Documentation
Navigator SOPs:
- Complete Release Workflow:
.agent/sops/development/complete-release-workflow.md - Multi-Claude Workflow Guide:
scripts/POC-LEARNINGS.md
Code References:
- Orchestrator:
scripts/navigator-multi-claude.sh - POC Script:
scripts/navigator-multi-claude-poc.sh - Monitor:
scripts/sub-claude-monitor.sh - Resume:
scripts/resume-workflow.sh
External:
- Claude Code CLI docs: https://docs.claude.com
- Navigator GitHub: https://github.com/alekspetrov/navigator
Maintenance
Update this SOP when:
- New failure mode discovered
- Recovery procedure improved
- Workflow scripts modified
- Timeout defaults changed
Owner: Navigator maintainers Review frequency: After major releases or significant workflow failures
Last Updated: 2025-11-02 Tested With: v4.5.0 (TASK-25 implementation, dogfooding) Known Issues: Monitor timeout detection needs refinement for long-running phases