Multi Agent Orchestration

Overview

Single agents are limited by context window, specialization depth, and parallelism. Multi-agent systems overcome these limits by routing subtasks to specialized agents. But multi-agent systems introduce new failure modes: lost context, conflicting decisions, infinite loops, and cascading failures.

This skill provides the architecture and coordination patterns to build multi-agent systems that are reliable, observable, and maintainable.

When to Use

• The task requires more context than a single agent can handle
• Different subtasks require different specializations (research, coding, review, security)
• Subtasks can be parallelized for speed
• The workflow is long-running and requires checkpointing
• Different tasks require different levels of human oversight

Process

Step 1: Design the Agent Network

1. Define agent responsibilities: Each agent should have a single, well-defined job. Name them by role: researcher, coder, reviewer, security-auditor, tester.
2. Define communication topology: Who can talk to whom?
   • Pipeline: Agent A → Agent B → Agent C (sequential)
   • Supervisor: Orchestrator dispatches to specialists (hub-and-spoke)
   • Peer: Agents collaborate as equals (mesh)
3. Define data contracts: What does each agent receive? What does it output? Use structured formats (JSON schemas) for inter-agent communication.
4. Define the orchestration logic: Who decides which agent acts next?

Verify: You can draw the agent network on a whiteboard with clear roles and data flow.

Step 2: Implement Context Management

5. Each agent should receive only the context it needs — not the full conversation history.
6. Use a shared state store (database, key-value store) for information that multiple agents need.
7. Pass summaries, not full transcripts, when context must traverse agent boundaries.
8. Include a task ID in every message for tracing.

Verify: No agent receives more context than it requires for its specific task.

Step 3: Design for Failure

9. Every agent call can fail — plan for it:
   • Timeout with a defined maximum duration
   • Retry with exponential backoff (max 3 retries)
   • Fallback behavior when retries are exhausted
10. Prevent infinite loops: Track call depth. If depth > N (e.g., 10), surface to human review.
11. Checkpointing: For long workflows, save state after each major step so the workflow can be resumed after failure.
12. Dead letter queue: Failed tasks that exhaust retries go to a queue for human inspection.

Verify: Failure scenarios are defined for every agent-to-agent call.

Step 4: Human-in-the-Loop Checkpoints

13. Define which decisions require human approval:
    • Irreversible actions (data deletion, financial transactions, external communications)
    • High-uncertainty states (agents disagree, confidence below threshold)
    • Sensitive operations (PII access, privileged system access)
14. Design the human review interface: What information does the reviewer need? What actions can they take?

Verify: At least one human-in-the-loop checkpoint exists for high-risk operations.

Step 5: Observability

15. Log every agent invocation: inputs, outputs, duration, token usage, errors.
16. Implement distributed tracing across the agent network (trace ID propagated through all calls).
17. Dashboard: agent activity, success/failure rates, latency, token consumption.
18. Alerts: agent down, retry rate spike, context overflow, unexpected output patterns.

Verify: You can trace any specific task's full execution path across all agents from logs alone.

Common Rationalizations (and Rebuttals)

Red Flags

• Agents pass full conversation history to other agents (context bloat)
• No timeout defined for any agent call
• Agents can call each other recursively without depth limits
• No human approval required for irreversible actions
• No distributed tracing across agent boundaries
• Agents making conflicting state changes with no conflict resolution

Verification

• Agent network designed with clear roles and data contracts
• Context is minimized at each agent boundary
• Failure handling (timeout, retry, fallback) for every agent call
• Infinite loop prevention via call depth limits
• Human-in-the-loop checkpoints for high-risk operations
• Distributed tracing implemented across agents
• End-to-end test of failure scenarios

References

• task-decomposition skill
• observability skill
• prompt-injection-defense skill
• hallucination-prevention skill