tomevault-io AI Agent Orchestration
Design patterns and production workflows for multi-agent systems. Sources: Microsoft Azure Architecture Center, Google AI Agent Clinic, Cyrus orchestration case study.
Complexity Spectrum
Before adopting multi-agent orchestration, evaluate whether the task requires it. Each level adds coordination overhead, latency, and cost.
| Level | When to Use | Escalation Trigger |
|---|---|---|
| Direct model call | Single-step tasks (classification, summarization, translation) | Task requires tool access or multi-step reasoning |
| Single agent with tools | Varied queries within one domain, dynamic tool use | Cross-domain problems, security boundaries, tool overload, prompt complexity |
| Multiagent orchestration | Cross-functional problems, distinct security per agent, tasks benefiting from parallel specialization | — |
Use the lowest complexity level that reliably meets requirements.
Five Orchestration Patterns
Decision Matrix
| Pattern | Coordination | Routing | Best For | Risk |
|---|---|---|---|---|
| Sequential | Linear pipeline, output flows A→B→C | Deterministic, predefined order | Step-by-step refinement, clear dependencies | Early-stage failures cascade; no parallelism |
| Concurrent | Parallel, agents work on same input independently | Deterministic or dynamic selection | Multiple perspectives, latency-sensitive tasks | Conflicting results need resolution; resource-intensive |
| Group Chat | Shared conversation thread, chat manager controls turns | Manager-mediated turn order | Consensus-building, brainstorming, maker-checker validation | Infinite conversation loops; hard to control with >3 agents |
| Handoff | One active agent at a time, dynamic delegation | Agents self-route based on context | Tasks where optimal specialist emerges during processing | Infinite handoff loops; unpredictable routing paths |
| Magentic | Manager builds dynamic task ledger, iterations refine plan | Manager assigns/reorders tasks adaptively | Open-ended problems with no predetermined solution path | Slow to converge; stalls on ambiguous goals |
Sequential Orchestration
When to use: Multistage processes with clear linear dependencies (draft→review→polish), data transformation pipelines, progressive refinement.
When to avoid: Stages can be parallelized, only 1-2 stages exist, early stages might fail with no recovery path, backtracking or iteration needed.
Example: Contract generation pipeline — template selection → clause customization → regulatory compliance → risk assessment. Each agent builds on the previous output.
Concurrent Orchestration
When to use: Tasks benefiting from multiple independent perspectives (technical, business, creative), time-sensitive parallel processing, ensemble reasoning or voting-based decisions.
When to avoid: Agents need cumulative context in sequence, no clear conflict resolution strategy, resource constraints make parallelism impossible.
Example: Stock analysis — fundamental analysis, technical analysis, sentiment analysis, and ESG evaluation run in parallel, then results are aggregated into a recommendation.
Group Chat Orchestration
When to use: Creative brainstorming requiring cross-functional dialogue, structured maker-checker quality gates, human-in-the-loop scenarios, compliance validation needing multiple expert perspectives.
When to avoid: Basic task delegation suffices, real-time processing required, no objective way to determine task completion, more than 3 agents (control degrades).
Maker-checker sub-pattern: One agent creates (maker), another evaluates against criteria (checker). Checker rejects with specific feedback → maker revises → cycle repeats until approval or iteration cap. Requires clear acceptance criteria and an iteration limit with defined fallback behavior.
Handoff Orchestration
When to use: Specialized knowledge or tools needed but the right agent isn't known upfront, expertise requirements emerge during processing, multi-domain problems needing different specialists sequentially.
When to avoid: Appropriate agent is identifiable from initial input (use deterministic routing instead), suboptimal routing could frustrate users, infinite handoff loops are hard to prevent.
Example: Customer support — triage agent handles common issues, hands off network problems to infrastructure agent, billing disputes to financial resolution agent. Agents can further hand off or escalate to human support.
Magentic Orchestration
When to use: Complex open-ended problems with no predetermined solution, requirement to generate a plan for human review before execution, agents equipped with tools that modify external systems.
When to avoid: Deterministic solution path exists, no plan ledger needed, low complexity, time-sensitive (this pattern optimizes for correctness not speed).
Example: SRE incident response — manager agent builds initial task ledger (restore service, identify root cause), consults diagnostics/infrastructure/rollback/communication agents, continuously refines the plan as new information emerges.
Implementation Guardrails
- Context management: Compact context between agents via summarization; persist shared state externally for long-running tasks.
- Reliability: Timeout and retry mechanisms, circuit breakers, validate agent output before forwarding, isolate agents from shared failure points.
- Security: Least privilege per agent, authenticate inter-agent communication, apply content safety guardrails at user input, tool calls, tool responses, and final output.
- Cost: Assign smaller/cheaper models to classification/extraction/formatting agents; monitor per-agent token consumption; compact context between agents.
- Observability: Instrument all agent operations and handoffs; use scoring rubrics or LLM-as-judge evaluations for testing nondeterministic agent outputs.
Five Production Hardening Lessons
From Google's AI Agent Clinic refactoring of "Titanium" — a sales research agent rebuilt from prototype to production.
1. Ditch the Monolith for Orchestrated Sub-Agents
| Anti-Pattern | Fix |
|---|---|
Monolithic script with linear for loop — one failure stalls everything |
Decompose into specialized agents in a SequentialAgent pipeline |
| Single LLM executing massive multi-step prompts | Narrow agents with single responsibilities |
Action: When a single agent's prompt exceeds ~200 lines or touches 3+ distinct domains, decompose into orchestrated sub-agents.
2. Force Structured Outputs with Schema Contracts
| Anti-Pattern | Fix |
|---|---|
| JSON format instructions embedded in prompt strings | Inject native schema objects (Pydantic, Zod, etc.) as explicit contracts |
| Brittle string parsing of model outputs | Framework enforces structural adherence at runtime |
Action: Replace any prompt string that says "return JSON in this format" with a typed schema. Let the framework handle serialization.
3. Replace Hardcoded State with Dynamic RAG Pipelines
| Anti-Pattern | Fix |
|---|---|
| Hardcoded data in source files (e.g., 12 case studies in Python) | Autonomous crawling pipeline feeding a vector search index |
| Manual code changes to update agent knowledge | Hybrid search (semantic + keyword) over indexed corpus |
Action: Any data list embedded in code that grows over time should be replaced with a RAG pipeline — crawl, embed, index, query.
4. Observability is Non-Negotiable
| Anti-Pattern | Fix |
|---|---|
| Black-box failures with no component-level diagnostics | OpenTelemetry distributed traces for full execution flows |
| No visibility into which agent caused a break | Live telemetry dashboard capturing model requests, tokens, and tool executions |
Action: Instrument with OpenTelemetry before deploying to production. Trace every agent transition, model call, and tool invocation.
5. Tame Token Burn with Circuit Breakers
| Anti-Pattern | Fix |
|---|---|
| Agent retries prompts without bounds on errors | Exponential backoff, timeout boundaries, configurable retry limits |
| Custom try-catch retry logic in application code | Let the orchestration framework handle graceful failures natively |
Action: Every agent invocation should have a max-retry ceiling, a timeout, and a defined fallback. Never allow unbounded retry loops.
Practical Orchestration Workflow
From the Cyrus case study: refactoring a complex webhook system (5,554 lines deprecated, 3 competing implementations, 59 files) in 3.5 hours with zero human intervention.
The 5-Step Loop
1. Atomic Decomposition
- Create precise, verifiable sub-tasks — not vague instructions.
- Bad: "Clean up the webhook code." Good: "Delete packages/ndjson-client/ directory. Remove all NdjsonClient imports from EdgeWorker. Verify 99 tests passing."
- Each sub-task must have concrete, binary acceptance criteria.
2. Sequential Dependency Management
- Identify and respect dependency order. Don't over-parallelize.
- Can't convert to a new framework until deprecated code is removed. Can't integrate until all components are refactored.
- Build a dependency graph and execute in dependency order.
3. Independent Verification Before Integration
- After each sub-agent completes: pull its output, run tests/typecheck/build independently.
- Do not trust self-reported success. Verify before merging into the main line.
- A bad merge in hour 1 cascades through hours 2–4.
4. Feedback Loops with Specific Diagnosis
- When verification fails, provide: specific failure count, affected files, root cause diagnosis, actionable fix list.
- Allow the sub-agent to correct and re-submit.
- Result: agent learns and delivers working code without human escalation.
5. Isolated Workspaces
- Each sub-agent works in a separate git worktree or sandbox.
- Prevents merge conflicts, corrupts nothing on failure, enables clean rollback.
- Orchestrator pulls completed work only after verification passes.
When to Use Orchestration vs. Not
| Use Orchestration | Don't Use Orchestration |
|---|---|
| Task has clear sub-problems with concrete acceptance criteria | Task is simple and linear (single-file changes) |
| Dependencies between sub-tasks exist | No dependencies to manage |
| Each sub-task can be independently verified (tests pass/fail is binary) | Requirements are ambiguous, need human judgment |
| Human coordination cost exceeds orchestration setup cost (tasks taking days+) | Task takes less than 1 hour of human time |
| You want a documented, reproducible process | You need creative problem-solving (novel algorithms, UX design) |
Verification
Verify this skill activates correctly:
- Present a task: "Refactor this monolithic agent into orchestrated sub-agents for our document processing pipeline."
- Confirm the agent identifies the complexity spectrum, selects an appropriate orchestration pattern, and applies the production hardening checklist.
- Present a task: "We need to choose between concurrent and group chat orchestration for our compliance review system."
- Confirm the agent uses the decision matrix to evaluate trade-offs and recommends a pattern with justification.
Source: pngdeity/apm-user-repository — distributed by TomeVault.