Use this skill when designing, auditing, or diagnosing an AI agent's system architecture. Activates on "agent architecture", "agent design", "agent audit", "multi-agent system", "agent orchestration", "agent topology", "agent evaluation design", or "agent execution flow". Covers agent topology design, reference architecture patterns, audit frameworks, evaluation design, execution continuity, and architectural decision records. Outputs architecture diagrams, RFC-style design docs, audit reports with severity-ranked findings, and migration plans.
An agent architecture is the set of decisions that determine what an agent can do, what it cannot do, what happens when it fails, and how you will find out. Most agent failures in production are not model failures. They are architecture failures: an unbounded loop with no stop condition, a supervisor that cannot tell a subagent's failure from its success, a tool contract that returns a plausible string where a typed error was needed, a state machine whose recovery path nobody ever executed.
This skill is for the design half and the audit half of that problem. It gives you a small set of topology patterns with real trade-offs, a design process that forces the expensive decisions before code (scope → topology → contract → eval → iterate), a severity-ranked audit framework, an ADR template so decisions survive their authors, and an evaluation design that measures the architecture rather than the demo.
The core stance: an agent is a distributed system with a probabilistic component inside it. Everything you know about distributed systems still applies — partial failure, idempotency, retries, observability, contract versioning — and the probabilistic component makes the failure modes worse, not different. Design for the failure, not the happy path.
When to Use
Designing a new agent or multi-agent system, or writing its design document
Auditing an existing agent before it gets production traffic, a new owner, or a bigger tool budget
Diagnosing an agent that "sometimes does the wrong thing" and nobody can reproduce it
Choosing a topology: single agent, supervisor, swarm, pipeline, mesh
Defining what an agent is allowed to do: tool surface, permissions, escalation, human gates
Designing the evaluation suite, or arguing about whether an eval proves anything
Writing the ADR that records why the architecture looks like this
Planning a migration from one topology to another
When not to use: tuning a prompt belongs to prompt-engineering; writing one system prompt or tool description belongs to prompt authoring; building the eval harness's scoring code belongs to agentic-eval; threat-modelling a specific attack surface belongs to threat-modeling. This skill owns the shape — who talks to whom, what the contract is, where state lives, how you know it works.
Agent Topology Patterns
Five patterns cover nearly everything. Pick deliberately and write down why; the default failure is arriving at swarm by accretion.
1. Single agent (with tools)
One model, one loop, a bounded tool set. The loop is: observe → decide → act → observe.
Use when: subtasks are heterogeneous (research vs. code vs. review), or workers need different tools/models/permissions.
Strengths: isolation (a worker's context pollution stays local), per-worker model choice, per-worker permission scoping.
Weaknesses: the supervisor becomes a bottleneck and a single point of misinterpretation; result synthesis is where quality dies silently.
Contract rules: every worker returns a typed envelope (status, output, error class, cost, confidence), never free text. The supervisor must distinguish "worker failed" from "worker returned nothing" from "worker returned the wrong shape".
3. Swarm (peer delegation)
Agents hand off control to peers; no fixed lead. Handoff is a tool call: transfer_to(agent, reason).
┌────────┐ ⇄ ┌────────┐
│Agent A │ │Agent B │
└───┬────┘ └────┬───┘
│ ⇄ ┌───────▼──┐
└────▶│ Agent C │
└──────────┘
Use when: the task path is genuinely unknown up front and each specialist knows best who should handle the next step (triage → billing → refund).
Strengths: no central bottleneck, natural for routing-heavy workloads.
Weaknesses: unbounded handoff loops (A→B→A), no global view, hard to attribute cost, hard to evaluate.
Contract rules: a global handoff budget (max N transfers) and a visited set so an agent cannot be re-entered without new information. Every handoff carries a serialized state object, not a conversation replay.
4. Pipeline (deterministic stages)
Fixed sequence of stages; each stage may be an agent, a tool, or plain code. Control flow is code, not model judgement.
Use when: the workflow is known and mostly stable; you want reproducibility, per-stage retries, and per-stage evaluation.
Strengths: debuggable, cacheable, each stage independently testable and replaceable, cheap to re-run one stage.
Weaknesses: brittle on inputs the designer didn't anticipate; the model cannot reroute.
Contract rules: every boundary is a typed artifact on disk or in a store, with a schema and a version. A stage must be idempotent given the same input artifact.
5. Mesh (shared blackboard / event bus)
Agents publish to and subscribe from shared state; coordination is emergent. No direct calls.
┌────────┐ ┌──────────────┐ ┌────────┐
│Agent A │─────▶│ Blackboard │◀─────│Agent B │
└────────┘ │ (task queue, │ └────────┘
│ facts, log) │
┌────────┐ └──────┬───────┘ ┌────────┐
│Agent C │◀────────────┘ │Agent D │
└────────┘ └────────┘
Use when: long-running, many independent work items, agents of different kinds contributing to one evolving artifact (research corpus, incident response).
Strengths: highest parallelism and extensibility; add an agent without touching others.
Weaknesses: emergent behavior is the hardest thing to debug; duplicate work; no clear completion condition; cost scales non-linearly.
Contract rules: the blackboard schema is the architecture — version it, validate writes, define claim/lease semantics so two agents cannot do the same work.
Choosing a topology
Signal
Single
Supervisor
Swarm
Pipeline
Mesh
Subtasks known at design time
✓
✓
✗
✓✓
✗
Needs parallelism
✗
✓
~
✓
✓✓
Needs different tools/models per step
~
✓✓
✓
✓
✓
Reproducibility required
✓
~
✗
✓✓
✗
Failure must be attributable to one component
✓✓
✓
✗
✓✓
✗
Task path unknown up front
✗
✓
✓✓
✗
~
Team is new to agents
✓✓
✓
✗
✓
✗
Start at the top-left and move right only under a named pressure. Write the pressure down — "we chose swarm because X" is an ADR; "we chose swarm because it's the modern pattern" is how you inherit a system nobody can debug.
Architecture Design Process
Five steps, in order. Each step produces an artifact; skipping one moves its cost into production.
Step 1 — Scope: what the agent may and may not do
Write the boundary before the diagram.
One-sentence job statement. If it needs "and", you have two agents or two tasks.
Inputs and outputs, with types. "Takes a support email, returns a categorized ticket with citations."
Explicit non-goals. What the agent must refuse, and what it must escalate. Non-goals prevent scope creep more effectively than any prompt.
Blast radius: what is the worst thing this agent can do with its tools? Name it, then decide whether it is acceptable, gated, or scoped away.
Success criteria, measurable. "85% of tickets categorized correctly with zero fabricated policy citations", not "works well".
Step 2 — Topology: who exists and who talks to whom
Pick a pattern from the table above, with the pressure that justifies it.
Draw the graph: nodes are agents/tools/stores, edges are calls. Label each edge with the contract (what is passed) and whether it is sync or async.
Mark every human gate: where a human must approve before an irreversible action.
Mark every persistence boundary: where state leaves memory and becomes durable.
Count the maximum number of hops from user input to final output. That number is your worst-case latency and cost multiplier; if you cannot state it, the topology is unbounded.
Step 3 — Contract: the interfaces, not the prompts
This is the step teams skip, and it is the one that decides whether the system is debuggable.
Typed envelopes, not prose. A subagent that answers in prose forces the parent to parse intent; a parent that parses intent invents it.
Absence of output is a distinct state from empty output and from failure. Model all three.
Side-effecting tools need idempotency keys. Agents retry; without keys you double-charge the customer.
Tool descriptions are contracts. Ambiguous descriptions are the single largest source of wrong-tool selection; write them like docstrings with preconditions and failure modes.
Every tool declares what it does NOT do. Negative space is where the model guesses.
Step 4 — Eval: how you will know it works
Design the eval at the same time as the architecture, not after (see Evaluation Design). At minimum, before writing implementation code:
20–50 golden tasks with expected outcomes, including at least 5 adversarial ones.
A failure taxonomy you expect, so the eval can measure the classes you fear.
A decision threshold: what pass rate ships, what blocks, what needs human review.
Step 5 — Iterate: instrument, then change one thing
Ship with tracing from day one. An agent without per-step spans is undebuggable at the second failure.
Change one architectural variable per iteration (topology, model, tool surface, prompt). Two at once and you learn nothing.
Record each iteration as an ADR amendment, not a Slack thread.
Design document skeleton (RFC style)
# RFC: <agent name>
## Status
Draft | In review | Accepted <date> | Superseded by <link>
## Problem
<the job, in one paragraph, with the measurable success criterion>
## Non-goals
- <what this agent must refuse>
- <what stays human>
## Proposed architecture
<topology diagram; node and edge table with contracts>
## State and persistence
<what is durable, where, schema, retention>
## Failure model
| Failure | Detection | Response | Owner |
|---|---|---|---|
## Security and permissions
<tool surface, credential scope, human gates, blast radius>
## Evaluation plan
<golden set, metrics, thresholds, cadence>
## Cost model
<per-task token and tool cost, expected volume, ceiling>
## Rollout and rollback
<phases, flags, kill switch, what "off" means>
## Open questions
## Decision record
<links to ADRs>
Audit Framework
An audit answers one question: would this system fail loudly or silently? Score four dimensions, then rank findings by severity so the report is actionable rather than a list.
Dimension 1 — Completeness
Does the design account for the whole lifecycle, or only the happy path?
Every loop has a stop condition that is not model judgement (max turns, budget, deadline)
Every failure has a defined response: retry, fallback, escalate, or fail closed
Absence, empty, and error are three distinct modelled states at every boundary
Every side-effecting action has an idempotency key and a rollback or compensating action
Human gates exist at every irreversible action (payment, delete, external send, deploy)
State that must survive a restart is persisted, and recovery is a tested path, not a hope
The maximum hop count and maximum cost are stated and enforced
Tool schemas validate outputs, not just inputs
Dimension 2 — Security
Tool surface is least privilege; no agent holds a credential broader than its job
Untrusted input (web content, user files, tool results, other agents' output) is never treated as instruction
The agent cannot widen its own permissions via a tool call
Prompt-injection reachability is mapped: which tools can an injected instruction invoke?
Exfiltration paths are enumerated: what can leave, to where, carrying what
Secrets are not in the context window, prompts, logs, or traces
Every tool call is attributable to a principal (user, service, agent) in the audit log
Destructive tools are either absent, gated, or dry-run-first
Dimension 3 — Observability
One trace per task, one span per step, with model, token counts, cost, and latency on each
Prompts, tool args, and tool results are captured (with redaction) and retrievable by trace id
Errors are classified, not just logged — you can count contract_violation separately from timeout
Outcome is recorded per task (success/failure/escalated), not just activity
You can answer, from data: which step fails most, which tool is slowest, what a task costs at p95
Alerts exist on outcome drift (success rate, escalation rate, cost per task) — not just on 5xx
Replay is possible from a stored trace without re-running side effects
Dimension 4 — Cost
Per-task cost is measured, not estimated, and has a stated ceiling
Budgets are enforced (hard stop), not merely monitored
Token growth is bounded: history trimming, summarization, or retrieval — not unbounded accumulation
Model tier matches task difficulty; no frontier model on classification
Loops have a cost-per-iteration ceiling and a total ceiling
Cacheable work is cached (retrieval, classification, deterministic sub-calls)
Cost is attributable per user/task/agent so an outlier is visible
Severity ranking
Severity
Definition
Response
S1 — Critical
Silent data loss, unbounded spend, or an ungated irreversible action; will cause an incident
Block release
S2 — High
Failure mode exists with no detection; you will learn from the customer
Fix before scale
S3 — Medium
Detectable but slow/expensive to diagnose; degrades MTTR
Schedule
S4 — Low
Naming, structure, documentation, minor inefficiency
Backlog
Finding format — one block per finding, no prose paragraphs:
### S2-03: Supervisor cannot distinguish worker timeout from empty result
- **Where:** `supervisor.py:112` → `worker_client.call()`
- **Evidence:** exception path returns `[]`; supervisor treats `[]` as "no findings"
- **Impact:** silent under-reporting; a failed research worker reports "nothing found"
- **Fix:** return typed envelope `{status: "timeout"}`; supervisor escalates on non-`ok`
- **Effort:** S
Audit report skeleton:
# Architecture audit: <agent> — <date>
## Verdict: <ship | ship with fixes | do not ship> / risk: <low|med|high>
## Scorecard
| Dimension | Findings (S1/S2/S3/S4) | Assessment |
## Findings (severity ordered)
## What the architecture gets right
## Recommended migration path
Include what it gets right. An audit that is only a defect list gets argued with; an audit that shows the reviewer understood the design gets acted on.
Architectural Decision Record (ADR)
ADRs are how architecture survives the people who chose it. One decision per file, immutable once accepted — supersede, never edit. Store in docs/adr/NNNN-slug.md.
# ADR-0007: Supervisor topology over swarm for ticket routing
- **Status:** Accepted (2026-11-14)
- **Deciders:** <names/roles>
- **Supersedes:** —
- **Superseded by:** —
## Context
Routing has 4 known intents and a fixed escalation policy. Volume is 40k/day.
Handoff loops would be unbounded and cost is per-hop. Two engineers own the system.
## Decision
Supervisor with three specialist workers. Handoff is supervisor-only; workers
cannot call each other. Worker results return as typed envelopes.
## Consequences
- Positive: bounded hop count (max 3), attributable cost per worker, one policy point.
- Negative: supervisor is a single point of misinterpretation; adding an intent means touching the router.
- Neutral: worker prompts become stable, so per-worker eval is possible.
## Alternatives considered
- **Swarm:** rejected — unbounded handoff, no per-hop cost ceiling, needs eval we don't have.
- **Pipeline:** rejected — intents are not ordered; classification is genuinely conditional.
## Signals that would reopen this
> 10k/day and >10 intents, or supervisor misroutes exceeding 5% of traffic.
Rules: an ADR states the pressure, not just the choice; it records rejected alternatives with reasons; it names signals that would reopen it — a decision with no reopening condition becomes dogma.
Anti-Patterns
Anti-pattern
Symptom
Why it fails
Instead
Swarm by default
No one can explain a run's path
Emergent control flow with no attribution
Justify topology with a named pressure; start single-agent
Prose handoffs
Parent "interprets" child output
Intent parsed out of text is invented
Typed envelopes with status/error class
Unbounded loop
Costs spike; runs never end
Stop condition is model judgement
Max turns + budget + deadline, all enforced
Agent as router for deterministic work
Slow, expensive, flaky classification
Model used where a rule or query suffices
Deterministic code for deterministic steps
God tool
One tool with a mode parameter
Model must guess the mode; failures are ambiguous
Separate tools with tight schemas
Context as memory
Quality decays on long tasks
Window is finite and attention degrades
Durable store + retrieval; summarize deliberately
Silent fallback
Wrong answers, no errors
Degradation hidden behind a default
Fail loudly or return a typed degraded status
Eval by demo
"It worked when we tried it"
5 examples prove nothing about the tail
Golden set + adversarial set + thresholds
Prompt-only guardrails
Policy bypasses found in prod
Instructions are suggestions to a model
Tool-level and code-level enforcement
Agent owning its own permissions
Privilege creep
Self-modifying authority
Permissions fixed outside the loop
No persistence at step boundaries
Long tasks restart from zero
State lives in one context
Durable checkpoints after each stage
Retry without idempotency
Duplicate side effects
Agents retry on ambiguity
Idempotency keys on every mutation
Evaluation Design
The question an eval answers is not "does it work" but "which failure classes does it still have, at what rate, and is that acceptable?"
Layers
Layer
What it measures
Size
Cadence
Unit
One tool, one parser, one classifier
50–200 cases
Every commit
Step
One agent node's decision quality given inputs
20–50 per node
Every change to that node
Trajectory
The sequence of steps, not just the final answer
20–50 tasks
Every architecture change
End-to-end
Task outcome vs. expected outcome
50–200 tasks
Pre-release + nightly
Online
Real traffic outcomes, escalations, cost
continuous
Always
Trajectory evaluation is the one teams skip and the one that catches architecture bugs: an agent can reach the right answer through a path that cost 40x and would break on any perturbation. Score both outcome and path (hop count, tool errors, recoveries).
Golden set construction
Sampled from real traffic, not invented. Synthetic tasks share the designer's blind spots.
Stratified across intents, difficulty, and input length.
Expected outcome is checkable — exact match, schema validity, or a rubric a second model can apply reliably. "Looks good" is not an expectation.
Frozen and versioned. Silent edits to the golden set make every historical comparison a lie.
Metrics that matter
Metric
Definition
Why
Task success rate
Outcome matches expectation
The headline, and the least informative alone
Trajectory efficiency
Steps used / steps needed
Catches success-by-brute-force
Tool error rate
Failed calls / total calls
Separates tooling bugs from reasoning bugs
Recovery rate
Errors followed by a successful retry
Measures the failure model, not the model
Escalation rate
Tasks handed to a human
The honest cost signal; rising means the agent is failing politely
Harmful action rate
Ungated irreversible actions / tasks
Should be structurally zero
Cost per successful task
Total cost / successes
The number that decides viability
p95 latency
End-to-end
UX and timeout design
Pass criteria and the rules of eval
Threshold per layer, agreed before the run. Choosing a threshold after seeing results is how evals become theatre.
Report confidence. 50 tasks with 90% success has a wide interval; do not present it as 90%.
Track per-failure-class rates, not just aggregate. An aggregate that holds while escalations triple is a regression.
Every production incident grows the golden set. This is the only mechanism that makes the suite get better over time.
Pin the model version. An eval that changes when the provider ships is not a regression test.
Compare against a baseline, ideally the previous version and, for routing/classification work, a deterministic baseline.
Execution continuity
The eval and the audit share one question: can a run survive the middle?
Checkpoint after every stage that has a side effect or costs more than a few seconds.
A resumed run must not repeat completed side effects — checkpoint + idempotency key together.
Recovery is a tested path. Deliberately kill the process mid-run in staging and confirm the task completes correctly. An untested resume path is a myth.
Partial results are first-class. A run that fails at stage 4 of 5 must leave stages 1–3 usable, with a status that says so.
Timeout budgets compound. If each step has a 30s timeout and the task has 10 steps, the caller's timeout must account for the sum, not the max.
Long runs need a heartbeat and an owner. A task with no progress marker is indistinguishable from a hung one.
Red Flags
No stop condition other than "the model decides it's done"
Any irreversible tool without a human gate
Retries on a mutating tool with no idempotency key
A subagent whose failure is indistinguishable from an empty result
Cost measured monthly rather than per task
Traces that record prompts but not outcomes
Golden set written by the same person who wrote the prompts, from imagination
Topology chosen before the pressure that justifies it
An architecture with no ADR — or an ADR with no rejected alternatives
Numbers in the design doc with no measurement behind them
Verification
Before calling a design complete:
One-sentence job statement, typed inputs/outputs, and explicit non-goals
Topology chosen with the pressure written down, and an ADR recording rejected alternatives
Every node boundary has input schema, output schema, error classes, timeout, retry policy, budget, version
Maximum hop count and maximum cost per task stated and enforced
Every irreversible action gated or absent; blast radius named
Trace per task, span per step, outcome recorded, alerts on outcome drift
Golden set ≥20 tasks including adversarial cases; thresholds agreed before the run
Resume path exists, checkpointed, and tested by killing a run mid-flight
Rollout phased with a kill switch, and "off" defined
Before calling an audit complete:
All four dimensions scored with evidence, not impressions
Every finding has location, evidence, impact, fix, and severity
Findings severity-ordered; S1s block the verdict explicitly
Report states what the architecture gets right
Migration path given for S1/S2 findings, not just a defect list
1---2name: ai-agent-architecture3description: >-4 Use this skill when designing, auditing, or diagnosing an AI agent's system5 architecture. Activates on "agent architecture", "agent design", "agent audit",6 "multi-agent system", "agent orchestration", "agent topology", "agent7 evaluation design", or "agent execution flow". Covers agent topology design,8 reference architecture patterns, audit frameworks, evaluation design,9 execution continuity, and architectural decision records. Outputs architecture10 diagrams, RFC-style design docs, audit reports with severity-ranked findings,11 and migration plans.12version: 1.0.013author: Skill Foundry14license: MIT15category: ai-agent-engineering16compatibility: >-17 Cross-platform: Claude Code, OpenAI Codex, GitHub Copilot, Cursor, Windsurf,18 Gemini CLI, OpenClaw, Hermes Agent, and any SKILL.md-compatible agent.19tags:20 - agent-architecture21 - agent-design22 - multi-agent23 - orchestration24 - audit25 - evaluation26 - AI-agents27platforms:28 - claude-code29 - codex30 - cursor31 - gemini-cli32 - openclaw33 - copilot34 - windsurf35permissions:36 files.read: true37 files.write: true38 network: false39 shell.execute: false40 dangerous: false41metadata:42 source: https://skillmd.com/skills/jpeetz/ai-agent-architecture43 tags:44 - agent-architecture45 - agent-design46 - multi-agent47 - agent-audit48 - evaluation-design49 - topology50 related_skills:51 - agentic-eval52 - threat-modeling53 - prompt-engineering54---5556# AI Agent Architecture5758## Overview5960An agent architecture is the set of decisions that determine what an agent can do, what it cannot do, what happens when it fails, and how you will find out. Most agent failures in production are not model failures. They are architecture failures: an unbounded loop with no stop condition, a supervisor that cannot tell a subagent's failure from its success, a tool contract that returns a plausible string where a typed error was needed, a state machine whose recovery path nobody ever executed.6162This skill is for the design half and the audit half of that problem. It gives you a small set of topology patterns with real trade-offs, a design process that forces the expensive decisions before code (scope → topology → contract → eval → iterate), a severity-ranked audit framework, an ADR template so decisions survive their authors, and an evaluation design that measures the architecture rather than the demo.6364The core stance: **an agent is a distributed system with a probabilistic component inside it.** Everything you know about distributed systems still applies — partial failure, idempotency, retries, observability, contract versioning — and the probabilistic component makes the failure modes worse, not different. Design for the failure, not the happy path.6566## When to Use6768- Designing a new agent or multi-agent system, or writing its design document69- Auditing an existing agent before it gets production traffic, a new owner, or a bigger tool budget70- Diagnosing an agent that "sometimes does the wrong thing" and nobody can reproduce it71- Choosing a topology: single agent, supervisor, swarm, pipeline, mesh72- Defining what an agent is allowed to do: tool surface, permissions, escalation, human gates73- Designing the evaluation suite, or arguing about whether an eval proves anything74- Writing the ADR that records why the architecture looks like this75- Planning a migration from one topology to another7677**When not to use:** tuning a prompt belongs to `prompt-engineering`; writing one system prompt or tool description belongs to prompt authoring; building the eval harness's scoring code belongs to `agentic-eval`; threat-modelling a specific attack surface belongs to `threat-modeling`. This skill owns the *shape* — who talks to whom, what the contract is, where state lives, how you know it works.7879## Agent Topology Patterns8081Five patterns cover nearly everything. Pick deliberately and write down why; the default failure is arriving at swarm by accretion.8283### 1. Single agent (with tools)8485One model, one loop, a bounded tool set. The loop is: observe → decide → act → observe.8687```88┌─────────────────────────────┐89│ Controller (one loop) │90│ ┌────────┐ ┌──────────┐ │91│ │ Model │──▶│ Tool call│ │92│ └────▲───┘ └────┬─────┘ │93│ │ │ │94│ └── result ◀─┘ │95└─────────────────────────────┘96```9798- **Use when:** one clear job, tools under ~15, task fits in one context window, failure is cheap to retry.99- **Strengths:** simplest to debug, cheapest per task, one place for policy and logging.100- **Weaknesses:** context saturation on long tasks, no parallelism, tool selection degrades past ~15-20 tools.101- **Non-negotiable:** explicit stop condition (max turns), explicit budget (tokens/time/cost), explicit failure return.102103> Most systems that "need multi-agent" are one agent with a bad tool contract. Prove the single agent fails before adding a second one.104105### 2. Supervisor (router / orchestrator–worker)106107A lead agent decomposes the task, dispatches to specialists, and synthesizes their results. The supervisor is the only agent that talks to the user.108109```110 ┌────────────┐111 user ─────▶│ Supervisor │112 └──┬───┬───┬─┘113 ┌────────┘ │ └────────┐114 ▼ ▼ ▼115 ┌─────────┐ ┌─────────┐ ┌─────────┐116 │ Worker A│ │ Worker B│ │ Worker C│117 └─────────┘ └─────────┘ └─────────┘118 └────────── results ─────────┘119```120121- **Use when:** subtasks are heterogeneous (research vs. code vs. review), or workers need different tools/models/permissions.122- **Strengths:** isolation (a worker's context pollution stays local), per-worker model choice, per-worker permission scoping.123- **Weaknesses:** the supervisor becomes a bottleneck and a single point of misinterpretation; result synthesis is where quality dies silently.124- **Contract rules:** every worker returns a **typed envelope** (status, output, error class, cost, confidence), never free text. The supervisor must distinguish "worker failed" from "worker returned nothing" from "worker returned the wrong shape".125126### 3. Swarm (peer delegation)127128Agents hand off control to peers; no fixed lead. Handoff is a tool call: `transfer_to(agent, reason)`.129130```131 ┌────────┐ ⇄ ┌────────┐132 │Agent A │ │Agent B │133 └───┬────┘ └────┬───┘134 │ ⇄ ┌───────▼──┐135 └────▶│ Agent C │136 └──────────┘137```138139- **Use when:** the task path is genuinely unknown up front and each specialist knows best who should handle the next step (triage → billing → refund).140- **Strengths:** no central bottleneck, natural for routing-heavy workloads.141- **Weaknesses:** unbounded handoff loops (A→B→A), no global view, hard to attribute cost, hard to evaluate.142- **Contract rules:** a **global handoff budget** (max N transfers) and a **visited set** so an agent cannot be re-entered without new information. Every handoff carries a serialized state object, not a conversation replay.143144### 4. Pipeline (deterministic stages)145146Fixed sequence of stages; each stage may be an agent, a tool, or plain code. Control flow is code, not model judgement.147148```149extract ──▶ classify ──▶ plan ──▶ execute ──▶ verify150 │ │ │ │ │151 typed typed typed typed typed152 output output output output gate153```154155- **Use when:** the workflow is known and mostly stable; you want reproducibility, per-stage retries, and per-stage evaluation.156- **Strengths:** debuggable, cacheable, each stage independently testable and replaceable, cheap to re-run one stage.157- **Weaknesses:** brittle on inputs the designer didn't anticipate; the model cannot reroute.158- **Contract rules:** every boundary is a **typed artifact** on disk or in a store, with a schema and a version. A stage must be idempotent given the same input artifact.159160### 5. Mesh (shared blackboard / event bus)161162Agents publish to and subscribe from shared state; coordination is emergent. No direct calls.163164```165 ┌────────┐ ┌──────────────┐ ┌────────┐166 │Agent A │─────▶│ Blackboard │◀─────│Agent B │167 └────────┘ │ (task queue, │ └────────┘168 │ facts, log) │169 ┌────────┐ └──────┬───────┘ ┌────────┐170 │Agent C │◀────────────┘ │Agent D │171 └────────┘ └────────┘172```173174- **Use when:** long-running, many independent work items, agents of different kinds contributing to one evolving artifact (research corpus, incident response).175- **Strengths:** highest parallelism and extensibility; add an agent without touching others.176- **Weaknesses:** emergent behavior is the hardest thing to debug; duplicate work; no clear completion condition; cost scales non-linearly.177- **Contract rules:** the blackboard schema **is** the architecture — version it, validate writes, define claim/lease semantics so two agents cannot do the same work.178179### Choosing a topology180181| Signal | Single | Supervisor | Swarm | Pipeline | Mesh |182|---|---|---|---|---|---|183| Subtasks known at design time | ✓ | ✓ | ✗ | ✓✓ | ✗ |184| Needs parallelism | ✗ | ✓ | ~ | ✓ | ✓✓ |185| Needs different tools/models per step | ~ | ✓✓ | ✓ | ✓ | ✓ |186| Reproducibility required | ✓ | ~ | ✗ | ✓✓ | ✗ |187| Failure must be attributable to one component | ✓✓ | ✓ | ✗ | ✓✓ | ✗ |188| Task path unknown up front | ✗ | ✓ | ✓✓ | ✗ | ~ |189| Team is new to agents | ✓✓ | ✓ | ✗ | ✓ | ✗ |190191Start at the top-left and move right only under a named pressure. Write the pressure down — "we chose swarm because X" is an ADR; "we chose swarm because it's the modern pattern" is how you inherit a system nobody can debug.192193## Architecture Design Process194195Five steps, in order. Each step produces an artifact; skipping one moves its cost into production.196197### Step 1 — Scope: what the agent may and may not do198199Write the boundary before the diagram.200201- **One-sentence job statement.** If it needs "and", you have two agents or two tasks.202- **Inputs and outputs**, with types. "Takes a support email, returns a categorized ticket with citations."203- **Explicit non-goals.** What the agent must refuse, and what it must escalate. Non-goals prevent scope creep more effectively than any prompt.204- **Blast radius:** what is the worst thing this agent can do with its tools? Name it, then decide whether it is acceptable, gated, or scoped away.205- **Success criteria, measurable.** "85% of tickets categorized correctly with zero fabricated policy citations", not "works well".206207### Step 2 — Topology: who exists and who talks to whom208209- Pick a pattern from the table above, with the pressure that justifies it.210- Draw the graph: nodes are agents/tools/stores, edges are calls. Label each edge with the **contract** (what is passed) and whether it is **sync or async**.211- Mark every **human gate**: where a human must approve before an irreversible action.212- Mark every **persistence boundary**: where state leaves memory and becomes durable.213- Count the **maximum number of hops** from user input to final output. That number is your worst-case latency and cost multiplier; if you cannot state it, the topology is unbounded.214215### Step 3 — Contract: the interfaces, not the prompts216217This is the step teams skip, and it is the one that decides whether the system is debuggable.218219For every node boundary define:220221| Element | Required | Example |222|---|---|---|223| Input schema | yes | `{ticket_id: str, body: str, locale: enum}` |224| Output schema | yes | `{category: enum, confidence: float, citations: list[Citation]}` |225| Error classes | yes | `retryable`, `terminal`, `needs_human`, `contract_violation` |226| Idempotency key | for any side effect | `hash(ticket_id, step)` |227| Timeout + retry policy | yes | 30s, 2 retries, exponential backoff, jitter |228| Token/cost budget | yes | 50k tokens, $0.10 per invocation |229| Version | yes | `categorizer/v2` |230231Rules that prevent most agent bugs:232233- **Typed envelopes, not prose.** A subagent that answers in prose forces the parent to parse intent; a parent that parses intent invents it.234- **Absence of output is a distinct state** from empty output and from failure. Model all three.235- **Side-effecting tools need idempotency keys.** Agents retry; without keys you double-charge the customer.236- **Tool descriptions are contracts.** Ambiguous descriptions are the single largest source of wrong-tool selection; write them like docstrings with preconditions and failure modes.237- **Every tool declares what it does NOT do.** Negative space is where the model guesses.238239### Step 4 — Eval: how you will know it works240241Design the eval at the same time as the architecture, not after (see [Evaluation Design](#evaluation-design)). At minimum, before writing implementation code:242243- 20–50 **golden tasks** with expected outcomes, including at least 5 adversarial ones.244- A **failure taxonomy** you expect, so the eval can measure the classes you fear.245- A **decision threshold**: what pass rate ships, what blocks, what needs human review.246247### Step 5 — Iterate: instrument, then change one thing248249- Ship with tracing from day one. An agent without per-step spans is undebuggable at the second failure.250- Change one architectural variable per iteration (topology, model, tool surface, prompt). Two at once and you learn nothing.251- Record each iteration as an ADR amendment, not a Slack thread.252253### Design document skeleton (RFC style)254255```markdown256# RFC: <agent name>257258## Status259Draft | In review | Accepted <date> | Superseded by <link>260261## Problem262<the job, in one paragraph, with the measurable success criterion>263264## Non-goals265- <what this agent must refuse>266- <what stays human>267268## Proposed architecture269<topology diagram; node and edge table with contracts>270271## State and persistence272<what is durable, where, schema, retention>273274## Failure model275| Failure | Detection | Response | Owner |276|---|---|---|---|277278## Security and permissions279<tool surface, credential scope, human gates, blast radius>280281## Evaluation plan282<golden set, metrics, thresholds, cadence>283284## Cost model285<per-task token and tool cost, expected volume, ceiling>286287## Rollout and rollback288<phases, flags, kill switch, what "off" means>289290## Open questions291## Decision record292<links to ADRs>293```294295## Audit Framework296297An audit answers one question: **would this system fail loudly or silently?** Score four dimensions, then rank findings by severity so the report is actionable rather than a list.298299### Dimension 1 — Completeness300301Does the design account for the whole lifecycle, or only the happy path?302303- [ ] Every loop has a **stop condition** that is not model judgement (max turns, budget, deadline)304- [ ] Every failure has a defined **response**: retry, fallback, escalate, or fail closed305- [ ] **Absence, empty, and error** are three distinct modelled states at every boundary306- [ ] Every side-effecting action has an **idempotency key** and a **rollback or compensating action**307- [ ] Human gates exist at every **irreversible** action (payment, delete, external send, deploy)308- [ ] State that must survive a restart is **persisted**, and recovery is a tested path, not a hope309- [ ] The **maximum hop count** and **maximum cost** are stated and enforced310- [ ] Tool schemas validate outputs, not just inputs311312### Dimension 2 — Security313314- [ ] Tool surface is **least privilege**; no agent holds a credential broader than its job315- [ ] Untrusted input (web content, user files, tool results, other agents' output) is **never** treated as instruction316- [ ] The agent cannot **widen its own permissions** via a tool call317- [ ] Prompt-injection reachability is mapped: which tools can an injected instruction invoke?318- [ ] Exfiltration paths are enumerated: what can leave, to where, carrying what319- [ ] Secrets are not in the context window, prompts, logs, or traces320- [ ] Every tool call is **attributable to a principal** (user, service, agent) in the audit log321- [ ] Destructive tools are either absent, gated, or dry-run-first322323### Dimension 3 — Observability324325- [ ] One **trace per task**, one **span per step**, with model, token counts, cost, and latency on each326- [ ] Prompts, tool args, and tool results are captured (with redaction) and **retrievable by trace id**327- [ ] Errors are classified, not just logged — you can count `contract_violation` separately from `timeout`328- [ ] **Outcome** is recorded per task (success/failure/escalated), not just activity329- [ ] You can answer, from data: which step fails most, which tool is slowest, what a task costs at p95330- [ ] Alerts exist on **outcome drift** (success rate, escalation rate, cost per task) — not just on 5xx331- [ ] Replay is possible from a stored trace without re-running side effects332333### Dimension 4 — Cost334335- [ ] Per-task cost is **measured**, not estimated, and has a stated ceiling336- [ ] Budgets are **enforced** (hard stop), not merely monitored337- [ ] Token growth is bounded: history trimming, summarization, or retrieval — not unbounded accumulation338- [ ] Model tier matches task difficulty; no frontier model on classification339- [ ] Loops have a cost-per-iteration ceiling and a total ceiling340- [ ] Cacheable work is cached (retrieval, classification, deterministic sub-calls)341- [ ] Cost is attributable per user/task/agent so an outlier is visible342343### Severity ranking344345| Severity | Definition | Response |346|---|---|---|347| **S1 — Critical** | Silent data loss, unbounded spend, or an ungated irreversible action; will cause an incident | Block release |348| **S2 — High** | Failure mode exists with no detection; you will learn from the customer | Fix before scale |349| **S3 — Medium** | Detectable but slow/expensive to diagnose; degrades MTTR | Schedule |350| **S4 — Low** | Naming, structure, documentation, minor inefficiency | Backlog |351352Finding format — one block per finding, no prose paragraphs:353354```markdown355### S2-03: Supervisor cannot distinguish worker timeout from empty result356- **Where:** `supervisor.py:112` → `worker_client.call()`357- **Evidence:** exception path returns `[]`; supervisor treats `[]` as "no findings"358- **Impact:** silent under-reporting; a failed research worker reports "nothing found"359- **Fix:** return typed envelope `{status: "timeout"}`; supervisor escalates on non-`ok`360- **Effort:** S361```362363Audit report skeleton:364365```markdown366# Architecture audit: <agent> — <date>367## Verdict: <ship | ship with fixes | do not ship> / risk: <low|med|high>368## Scorecard369| Dimension | Findings (S1/S2/S3/S4) | Assessment |370## Findings (severity ordered)371## What the architecture gets right372## Recommended migration path373```374375Include what it gets right. An audit that is only a defect list gets argued with; an audit that shows the reviewer understood the design gets acted on.376377## Architectural Decision Record (ADR)378379ADRs are how architecture survives the people who chose it. One decision per file, immutable once accepted — supersede, never edit. Store in `docs/adr/NNNN-slug.md`.380381```markdown382# ADR-0007: Supervisor topology over swarm for ticket routing383384- **Status:** Accepted (2026-11-14)385- **Deciders:** <names/roles>386- **Supersedes:** —387- **Superseded by:** —388389## Context390Routing has 4 known intents and a fixed escalation policy. Volume is 40k/day.391Handoff loops would be unbounded and cost is per-hop. Two engineers own the system.392393## Decision394Supervisor with three specialist workers. Handoff is supervisor-only; workers395cannot call each other. Worker results return as typed envelopes.396397## Consequences398- Positive: bounded hop count (max 3), attributable cost per worker, one policy point.399- Negative: supervisor is a single point of misinterpretation; adding an intent means touching the router.400- Neutral: worker prompts become stable, so per-worker eval is possible.401402## Alternatives considered403- **Swarm:** rejected — unbounded handoff, no per-hop cost ceiling, needs eval we don't have.404- **Pipeline:** rejected — intents are not ordered; classification is genuinely conditional.405406## Signals that would reopen this407> 10k/day and >10 intents, or supervisor misroutes exceeding 5% of traffic.408```409410Rules: an ADR states the **pressure**, not just the choice; it records **rejected alternatives with reasons**; it names **signals that would reopen it** — a decision with no reopening condition becomes dogma.411412## Anti-Patterns413414| Anti-pattern | Symptom | Why it fails | Instead |415|---|---|---|---|416| **Swarm by default** | No one can explain a run's path | Emergent control flow with no attribution | Justify topology with a named pressure; start single-agent |417| **Prose handoffs** | Parent "interprets" child output | Intent parsed out of text is invented | Typed envelopes with status/error class |418| **Unbounded loop** | Costs spike; runs never end | Stop condition is model judgement | Max turns + budget + deadline, all enforced |419| **Agent as router for deterministic work** | Slow, expensive, flaky classification | Model used where a rule or query suffices | Deterministic code for deterministic steps |420| **God tool** | One tool with a `mode` parameter | Model must guess the mode; failures are ambiguous | Separate tools with tight schemas |421| **Context as memory** | Quality decays on long tasks | Window is finite and attention degrades | Durable store + retrieval; summarize deliberately |422| **Silent fallback** | Wrong answers, no errors | Degradation hidden behind a default | Fail loudly or return a typed degraded status |423| **Eval by demo** | "It worked when we tried it" | 5 examples prove nothing about the tail | Golden set + adversarial set + thresholds |424| **Prompt-only guardrails** | Policy bypasses found in prod | Instructions are suggestions to a model | Tool-level and code-level enforcement |425| **Agent owning its own permissions** | Privilege creep | Self-modifying authority | Permissions fixed outside the loop |426| **No persistence at step boundaries** | Long tasks restart from zero | State lives in one context | Durable checkpoints after each stage |427| **Retry without idempotency** | Duplicate side effects | Agents retry on ambiguity | Idempotency keys on every mutation |428429## Evaluation Design430431The question an eval answers is not "does it work" but "**which failure classes does it still have, at what rate, and is that acceptable?**"432433### Layers434435| Layer | What it measures | Size | Cadence |436|---|---|---|---|437| **Unit** | One tool, one parser, one classifier | 50–200 cases | Every commit |438| **Step** | One agent node's decision quality given inputs | 20–50 per node | Every change to that node |439| **Trajectory** | The sequence of steps, not just the final answer | 20–50 tasks | Every architecture change |440| **End-to-end** | Task outcome vs. expected outcome | 50–200 tasks | Pre-release + nightly |441| **Online** | Real traffic outcomes, escalations, cost | continuous | Always |442443Trajectory evaluation is the one teams skip and the one that catches architecture bugs: an agent can reach the right answer through a path that cost 40x and would break on any perturbation. Score both **outcome** and **path** (hop count, tool errors, recoveries).444445### Golden set construction446447- **Sampled from real traffic**, not invented. Synthetic tasks share the designer's blind spots.448- **Stratified** across intents, difficulty, and input length.449- **Adversarial slice**, minimum 20%: prompt injection, missing fields, contradictory instructions, tool failures, empty results, out-of-scope requests.450- **Expected outcome is checkable** — exact match, schema validity, or a rubric a second model can apply reliably. "Looks good" is not an expectation.451- **Frozen and versioned.** Silent edits to the golden set make every historical comparison a lie.452453### Metrics that matter454455| Metric | Definition | Why |456|---|---|---|457| Task success rate | Outcome matches expectation | The headline, and the least informative alone |458| Trajectory efficiency | Steps used / steps needed | Catches success-by-brute-force |459| Tool error rate | Failed calls / total calls | Separates tooling bugs from reasoning bugs |460| Recovery rate | Errors followed by a successful retry | Measures the failure model, not the model |461| Escalation rate | Tasks handed to a human | The honest cost signal; rising means the agent is failing politely |462| Harmful action rate | Ungated irreversible actions / tasks | Should be structurally zero |463| Cost per successful task | Total cost / successes | The number that decides viability |464| p95 latency | End-to-end | UX and timeout design |465466### Pass criteria and the rules of eval467468- **Threshold per layer, agreed before the run.** Choosing a threshold after seeing results is how evals become theatre.469- **Report confidence.** 50 tasks with 90% success has a wide interval; do not present it as 90%.470- **Track per-failure-class rates, not just aggregate.** An aggregate that holds while escalations triple is a regression.471- **Every production incident grows the golden set.** This is the only mechanism that makes the suite get better over time.472- **Pin the model version.** An eval that changes when the provider ships is not a regression test.473- **Compare against a baseline**, ideally the previous version and, for routing/classification work, a deterministic baseline.474475### Execution continuity476477The eval and the audit share one question: can a run survive the middle?478479- **Checkpoint after every stage** that has a side effect or costs more than a few seconds.480- **A resumed run must not repeat completed side effects** — checkpoint + idempotency key together.481- **Recovery is a tested path.** Deliberately kill the process mid-run in staging and confirm the task completes correctly. An untested resume path is a myth.482- **Partial results are first-class.** A run that fails at stage 4 of 5 must leave stages 1–3 usable, with a status that says so.483- **Timeout budgets compound.** If each step has a 30s timeout and the task has 10 steps, the caller's timeout must account for the sum, not the max.484- **Long runs need a heartbeat and an owner.** A task with no progress marker is indistinguishable from a hung one.485486## Red Flags487488- No stop condition other than "the model decides it's done"489- Any irreversible tool without a human gate490- Retries on a mutating tool with no idempotency key491- A subagent whose failure is indistinguishable from an empty result492- Cost measured monthly rather than per task493- Traces that record prompts but not outcomes494- Golden set written by the same person who wrote the prompts, from imagination495- Topology chosen before the pressure that justifies it496- An architecture with no ADR — or an ADR with no rejected alternatives497- Numbers in the design doc with no measurement behind them498499## Verification500501Before calling a design complete:502503- [ ] One-sentence job statement, typed inputs/outputs, and explicit non-goals504- [ ] Topology chosen with the pressure written down, and an ADR recording rejected alternatives505- [ ] Every node boundary has input schema, output schema, error classes, timeout, retry policy, budget, version506- [ ] Maximum hop count and maximum cost per task stated and enforced507- [ ] Every irreversible action gated or absent; blast radius named508- [ ] Trace per task, span per step, outcome recorded, alerts on outcome drift509- [ ] Golden set ≥20 tasks including adversarial cases; thresholds agreed before the run510- [ ] Resume path exists, checkpointed, and tested by killing a run mid-flight511- [ ] Rollout phased with a kill switch, and "off" defined512513Before calling an audit complete:514515- [ ] All four dimensions scored with evidence, not impressions516- [ ] Every finding has location, evidence, impact, fix, and severity517- [ ] Findings severity-ordered; S1s block the verdict explicitly518- [ ] Report states what the architecture gets right519- [ ] Migration path given for S1/S2 findings, not just a defect list
Run npx get-skill add jpeetz/ai-agent-architecture in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.
Use this skill when designing, auditing, or diagnosing an AI agent's system architecture. Activates on "agent architecture", "agent design", "agent audit", "multi-agent system", "agent orchestration", "agent topology", "agent evaluation design", or "agent execution flow". Covers agent topology design, reference architecture patterns, audit frameworks, evaluation design, execution continuity, and architectural decision records. Outputs architecture diagrams, RFC-style design docs, audit reports with severity-ranked findings, and migration plans. It is listed under AI & ML on SkillMD.
This skill has not completed SkillMD's automated safety review yet. Capability flags: reads secrets. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.
This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.
Yes. Installing skills from SkillMD is free. This skill is licensed under MIT.
JPeetz (@jpeetz) published this skill. Their other Agent Skills are listed on their SkillMD profile.