Agent Squad Swift

AgentSquad Swift — assistant guide

Protocol-driven, on-device multi-agent framework (Swift 6.2, iOS 16+ / macOS 14+; persistence via FileChatStorage on iOS 16+, DeviceChatStorage on iOS 17+). This is guidance and a map — not an API reference. Read the exact signatures from the source (swift/Sources/AgentSquad/) and the worked recipes from the docs site sources (docs/src/content/docs/swift/); this file tells you what to use, when, and what to watch out for.

When to use what

• One assistant → an Agent (or GroundedAgent) + an Orchestrator with no classifier. No routing hop.
• Several specialists → multiple agents + an LLMClassifier; the orchestrator routes each turn.
• Answers must not drift from data (prices, stock, balances) → GroundedAgent: a Brain calls tools, an isolated Presenter speaks only from the curated results (it can be a smaller/local model).
• Voice → a VoiceAssistant (a peer of the orchestrator, not an agent): OpenAIVoiceAssistant (single LLM + tools, speaks directly — the spoken analog of Agent) or OpenAIGroundedVoiceAssistant (Brain → Presenter, can't drift from data — analog of GroundedAgent).

Every component is a Sendable protocol with one built-in implementation — swap in your own anywhere.

Modules (import only what you use)

SwiftPM: .package(url: "https://github.com/2FastLabs/agent-squad", branch: "main").

How a turn works

Two peer runtimes share contracts but not a control loop: a turn-based Orchestrator (classify? → run agent → stream → persist) and a long-lived VoiceAssistant for voice. Either way you consume an AsyncThrowingStream<AgentEvent, any Error> — the one idiom worth memorizing:

for try await event in orchestrator.route(.text("hello"), userId: "u1", sessionId: "s1") {
    switch event {
    case .textDelta(let token): /* stream tokens */
    case .final(let message):   /* the message that was persisted */
    case .toolCall, .widget, .thinking, .error: break   // .error is a user-facing string
    }
}

.error carries a user-facing message; real programmer/transport failures throw through the stream. .final is what the orchestrator persists. Inputs/messages are value types (AgentInput.text, ConversationMessage, ContentPart, JSONValue) in Sources/AgentSquad/Core/.

The pieces

• Orchestrator drives a turn. The classifier is optional — omit it for a single agent.
• Agent is one LLM with an internal tool loop. GroundedAgent is two LLMs (Brain + isolated Presenter) for answers that must stay grounded in tool results. The Presenter never sees chat history or the Brain's transcript; presenterInput picks .questionAndData (default) or .dataOnly.
• ChatCompletionsClient speaks the OpenAI wire — point its baseURL at OpenAI, Azure, OpenRouter, Groq, or a local Ollama/llama.cpp. Implement LLMClient for anything else.
• Tools come from a ToolProvider. Built-ins: ToolKit holds native tools — Tool.local (Swift closure) and Tool.http/Tool.get/.post (declarative HTTP, with a ToolParameter DSL so you don't hand-write JSON Schema); HTTPToolGroup(baseURL:…) declares one API's shared config once, then one line per endpoint; MCPServer(url:) connects an MCP server; and AggregateToolProvider composes any mix behind one seam; DakeraRetriever(namespace:…) is a ToolProvider backed by a self-hosted Dakera memory server — it exposes a search_memory tool for grounding (and a direct retrieve(_:) API), talking to Dakera's REST endpoint over URLSession with no extra dependency. A ToolResult is three-part: text → the model's context, structuredContent → curator/UI data, ui → an optional widget.
• FileChatStorage (JSON files, iOS 16+) and DeviceChatStorage (SwiftData, iOS 17+) persist history on-device; InMemoryChatStorage is a non-persistent, seedable single-conversation store. TransformingChatStorage wraps any store and runs a MessageTransform before each save (PII scrub / redact / drop) — reads pass through, message.mappingText { … } covers the text-only case. SummarizingChatStorage wraps any store and keeps agent context small: on the first fetch that exceeds triggerAt message pairs the user-supplied ChatSummarizer is called and the compressed result is held in an in-memory buffer; subsequent saves append to the buffer and recompress eagerly if needed; fetchAllChats is never intercepted so raw history stays available for analytics — the inner store is never written by the summarizer. OSLogTracer is the default tracer; wire ProcessingTracer + OTLPExporter to ship traces to Langfuse/LangSmith/Datadog/…
• Voice: two VoiceAssistants over a WebSocket — OpenAIVoiceAssistant (single LLM, speaks directly) and OpenAIGroundedVoiceAssistant (grounded Brain → Presenter). Both are self-sufficient (own tracer/store/userId/sessionId; with a store, completed turns persist and prior history seeds on start()), wired to the mic/speaker by RealtimeRuntime. Preferred audio wiring: ONE VoiceProcessedAudioIO instance passed as both input: and output: — capture and playback share one voice-processed AVAudioEngine, so the assistant's audio is guaranteed to be in the echo canceller's reference path. The split MicCapture/AudioPlayback pair also works (capture is voice-processed by default; the AEC reference is then device-level/route-dependent; MicCapture(voiceProcessing: nil) = raw capture). All three audio classes take an AudioSessionPolicy (.managed / .custom / .external for apps that own the AVAudioSession) and a configureEngine hook exposing the raw AVAudioEngine. Session tuning on both: transcriptionModel (the user's STT only), turnDetection (.semanticVAD(eagerness:) / .serverVAD(threshold:…) / .disabled), and sessionOverrides (deep-merged into the generated session.update last — the escape hatch for unmodeled keys like audio.input.noise_reduction). On OpenAIVoiceAssistant additionally reasoning: RealtimeReasoningEffort (.minimal….xhigh, session-wide, reasoning models like gpt-realtime-2 only) and toolReasoningEffort: [String: RealtimeReasoningEffort] — turn-sticky escalation: once a turn calls a listed tool, that turn's subsequent responses are created with the mapped effort (highest wins), so payload synthesis thinks harder while lookups stay fast.

Custom implementations

Conform to the protocol and pass your type where the built-in goes. Each seam has a worked example on its doc page (paths below are under docs/src/content/docs/swift/, published at /agent-squad/swift/…); signatures live in Sources/AgentSquad/.

Gotchas

• maxToolRounds: Agent/GroundedAgent default to 20; the AgentProtocol default is 1. A custom agent that injects tools but leaves 1 silently disables its tool loop.
• Classifier is optional: no classifier ⇒ no routing hop / no extra model call. A nil selection falls back to the default agent (no confidence threshold).
• Persistence: only turns ending in .final are saved.
• ChatCompletionsClient: retries only before the first event; some local runtimes reject stream_options/unknown body keys — override via extraBody.
• JSONValue: whole-number doubles decode to .int; carry large IDs as .string.
• Storage: FileChatStorage (JSON, iOS 16+, scopes per-call by userId/sessionId/agentId — e.g. sessionId to isolate per match) or DeviceChatStorage (SwiftData, iOS 17+, bound to one userId). Both default to Library/Caches (disposable). InMemoryChatStorage (iOS 16+) is non-persistent and holds one conversation — construct it empty or seeded with a prior conversation to load one into a session. Wrap any store in TransformingChatStorage to scrub/redact before persistence; prefer redacting over returning nil (dropping one side of an exchange can make the store skip its counterpart via the consecutive-same-role guard). Wrap any store in SummarizingChatStorage(wrapping:summarizer:triggerAt:keepLast:) to keep agent context small: the buffer activates lazily on the first qualifying fetch; once active, saves append to it and compress eagerly; fetchAllChats bypasses the buffer entirely.
• Tracing lifecycle: nothing drains the tracer for you — flush on background, shut down on termination. OSLogTracer logs no payloads. Redaction hashes ids + clips strings but does not pattern-scrub PII — supply a custom Redactor for that. A realtime answer generation (response/presenter) is backdated to its response.created receive-time via SpanHandle.generation(…, startedAt:), so the exported span carries the real call latency instead of a ~0 duration (the Realtime API sends no server-side timing). The overload defaults to stamping now, so custom SpanHandles need not implement it.
• Realtime is a peer runtime, not an agent; its events stream is non-throwing; needs NSMicrophoneUsageDescription; always stop(). In-band failures arrive as .error(code:message:) — code is the API's machine code (e.g. rate_limit_exceeded), response_failed when the turn's live response ends with status: "failed" (a late failed done for a response already cancelled by barge-in is consumed silently), or transport_closed when the socket dies (then events finishes — the end-of-session signal); message is the human-readable detail for logs, not for verbatim display. Failures are recorded on the trace spans they end (turn/session/tool), so they export with status: error instead of vanishing.
• Barge-in truncation: on interrupt the session sends conversation.item.truncate so the server drops the unheard audio + transcript from context (OpenAI docs' WebSocket procedure). Automatic when wired by RealtimeRuntime with the built-in outputs; a custom AudioOutput gets it by implementing playedMilliseconds() (protocol default returns nil → truncation is skipped, everything else works). Applies to in-band spoken replies only — the grounded presenter is out-of-band (conversation: "none"), its items never enter the conversation, so interrupting it deliberately sends no truncate.
• Voice processing (AEC): on by default; if it can't be enabled start() throws .voiceProcessingUnavailable (degrade deliberately with MicCapture(voiceProcessing: nil)). For guaranteed echo cancellation use VoiceProcessedAudioIO and pass the same instance as input and output (its start()/stop() are idempotent — the runtime calls each twice). The simulator does no AEC — validate on a device. VP quiets the speaker (counter with duckingLevel: .min); never enable VP on a playback-only engine. With sessionPolicy: .external the app must configure and activate its AVAudioSession before start(), and should use the same policy everywhere.
• ContentPart Codable keys off case + label names — renaming breaks stored history.

Go deeper

• Prose & recipes — the Starlight docs under docs/src/content/docs/swift/ (run the site from docs/ with npm run dev): quick-start, orchestrator/overview, agents/built-in/grounded-agent, mcp/overview, ui/overview, storage/built-in/device, tracing/built-in/otlp-exporter, voice/built-in/openai-voice, voice/built-in/openai-grounded-voice, guides/*.
• Exact signatures — swift/Sources/AgentSquad/ (Core/, Agents/, Core/LLM/, Core/Tooling/, Core/Tracing/, Runtimes/Realtime/).