Persistent Agent Orchestration

persistent-agent-orchestration

Build persistence mechanisms before spawning autonomous agents to enable resumption and prevent orphaned processes.

Persistent Agent Orchestration

When spawning autonomous agents (especially Claude CLI agents), establish persistence infrastructure first to track agent state and enable recovery from failures.

Why This Matters

Without persistence:

• Agents may continue running if the parent process crashes
• No way to resume incomplete work
• Lost visibility into what agents were spawned and their status
• Difficult debugging and cleanup

Implementation Steps

1. Create a State File Structure

Before spawning any agents, define a state tracking system:

{
  "agent_id": "uuid-here",
  "status": "running|completed|failed",
  "created_at": "timestamp",
  "last_heartbeat": "timestamp",
  "output_path": "where results are stored",
  "pid": "process_id_if_applicable"
}

2. Generate Unique Identifiers

Assign each agent a UUID before spawning:

import uuid
agent_id = str(uuid.uuid4())
state_file = f"agents/{agent_id}.json"

3. Write State Before Spawning

Persist the initial state before launching the agent:

# Write "pending" state first
save_state(agent_id, {"status": "pending", "created_at": now()})
# Then spawn agent
spawn_agent(agent_id)

4. Implement Heartbeat/Checkpoint Logic

Have agents write periodic updates to their state file:

• Update last_heartbeat timestamp
• Log progress checkpoints
• Write partial results

5. Build Resumption Logic

On startup, scan for incomplete agents:

incomplete = [f for f in state_files if f["status"] != "completed"]
for agent_state in incomplete:
    resume_agent(agent_state["agent_id"])

Tips

• Store state files in a dedicated directory with predictable names
• Use atomic writes to prevent corruption
• Implement cleanup procedures for genuinely finished agents
• Consider a central registry if orchestrating many agents
• Add timeout logic to detect truly orphaned agents