LLM selection for agents
Resolve constraints in this sequence. Each step narrows the candidate set; never widen it later. Everything is over the ConvAI REST API (host https://api.elevenlabs.io, header xi-api-key: $API_KEY).
- Explore the agent. Read the current LLM, workflow, and tools with
GET /v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID first.
- Region (infrastructural, non-negotiable). Determine deployment topology. EU and India customers hit isolated deployments with restricted model lists that apply to all tiers. A model hidden by region cannot be enabled; do not offer to.
- Compliance entitlements (contract-gated). If the workspace requires HIPAA, PCI, or ZRM, filter to compliant models only. These are workspace-level flags managed by sales/CSMs, never self-serve.
- Capability fit (intelligence / latency / cost). Within whatever survives steps 1-3, recommend on task requirements.
If a constraint in step 2 or 3 eliminates the model the user wanted, say so plainly and route to their CSM or a support form. Do not present a capability recommendation that violates a hard constraint.
Step 1: region gate
| Region |
Rule |
| US / default |
Full model catalog, subject to compliance gates. |
| EU-isolated |
Restricted list. Preview Gemini models and several others are hidden. |
| India-isolated |
Heavily restricted: only custom-llm, glm-45-air-fp8, gpt-4o, qwen3-30b-a3b, qwen36-35b-a3b, speech-engine. |
Restriction is by deployment topology, not a toggle. Never offer to enable a region-hidden model.
Step 2: compliance gate
Only relevant if the workspace has the corresponding entitlement. You cannot see workspace flags, so never assert an entitlement is active; the safe line is "your CSM can confirm this for your workspace." Never offer to enable HIPAA, PCI, or ZRM; they require contract changes.
- HIPAA requires all three: enterprise tier with
force_logging_disabled=true, an LLM in the HIPAA-compliant set, and a signed BAA (a contract artifact you cannot verify; route BAA questions to sales). Approved families include Claude 3.5/3.7 Sonnet, Claude 3 Haiku, Claude Haiku 4.5, Claude Opus 4.7, Claude Sonnet 4 / 4.5 / 4.6, custom-llm, Gemini 1.5/2.0/2.5 flash and pro variants, gemini-3.1-flash-lite, and speech-engine. Models outside the list are not HIPAA-eligible regardless of tier; preview/GA-pending Gemini models are excluded.
- PCI is deny-by-default when on (
pci_compliance_required). Only pre-approved telephony providers, webhook domains, MCP URLs, and integration IDs work. Adding an integration requires CSM review; offer a support form, do not promise self-serve.
- ZRM disables audio/transcript storage; effective only when
force_logging_disabled=true. Custom LLM plus ZRM additionally needs is_convai_custom_llm_with_zrm_allowed. No Stripe SKU; part of the enterprise contract.
Step 3: capability fit
Recommend within the surviving candidate set. Weigh three axes against the agent's job.
- Latency matters most for voice. In an STT to LLM to TTS loop, LLM time-to-first-token sits on the critical path and is the dominant lever on perceived delay. For real-time agents, bias toward the fastest tier that clears the quality bar.
- Lowest latency:
gemini-3.1-flash-lite / nano / small qwen / *-mini tiers and speech-engine.
- Balanced:
gemini-3.5-flash, gpt-4o, gpt-5-mini, claude-haiku-4-5.
- Highest capability, higher latency:
claude-sonnet-4-6, claude-opus-4-7, gpt-5.x, gemini-3.x-pro (where region/compliance permits).
- Intelligence. Reserve the top tier for genuinely hard reasoning, complex multi-tool orchestration, or nuanced instruction-following. Most transactional voice agents (booking, triage, FAQ, routing) run well on a mid tier and feel snappier. Over-provisioning intelligence usually costs latency the user will notice for quality they will not.
- Cost. Per-token price scales steeply with capability tier. Match tier to task; do not put a frontier model behind a deterministic IVR flow. For high-concurrency deployments, latency and cost compound.
Quick heuristic
| Agent type |
Priority |
Typical pick (region/compliance permitting) |
| Real-time phone / high concurrency |
Latency, cost |
gemini-3.1-flash-lite (ultra-low latency) or gemini-3.5-flash |
| Complex reasoning / multi-tool |
Intelligence |
claude-sonnet-4-6, gpt-5.x, claude-opus-4-7 |
| HIPAA voice agent |
Compliance, then latency |
claude-haiku-4-5, gemini-3.5-flash (both in approved list) |
| India deployment |
Region first |
gpt-4o or qwen36-35b-a3b (catalog is tiny) |
| Custom/self-hosted model |
Control |
custom-llm (check ZRM + region eligibility) |
Apply the choice
Set the model with PATCH /v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID and body {"conversation_config":{"agent":{"llm":"<model-id>"}}}. This is branch-scoped and ships on merge.
Output discipline
- State the binding constraint explicitly ("In the EU deployment, X is not available, so among allowed models...").
- Give one primary recommendation plus a fallback, with the reason (latency vs intelligence vs cost).
- For anything contract-gated, end with the route: CSM for entitlement confirmation, a support form for new integrations/documents, and https://compliance.elevenlabs.io/ for attestations and signed DPA/BAA/sub-processor requests.
- Never claim an entitlement is active, never offer to enable a contract feature or a region-restricted model.
1---2name: architect-llm-selection3description: Use when choosing or recommending an LLM for an ElevenLabs agent, or answering "which model should this agent use", questions about region-restricted models, HIPAA/PCI/ZRM model eligibility, or latency/intelligence/cost tradeoffs between models.4---56# LLM selection for agents78Resolve constraints in this sequence. Each step narrows the candidate set; never widen it later. Everything is over the ConvAI REST API (host `https://api.elevenlabs.io`, header `xi-api-key: $API_KEY`).9101. Explore the agent. Read the current LLM, workflow, and tools with `GET /v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID` first.112. Region (infrastructural, non-negotiable). Determine deployment topology. EU and India customers hit isolated deployments with restricted model lists that apply to all tiers. A model hidden by region cannot be enabled; do not offer to.123. Compliance entitlements (contract-gated). If the workspace requires HIPAA, PCI, or ZRM, filter to compliant models only. These are workspace-level flags managed by sales/CSMs, never self-serve.134. Capability fit (intelligence / latency / cost). Within whatever survives steps 1-3, recommend on task requirements.1415If a constraint in step 2 or 3 eliminates the model the user wanted, say so plainly and route to their CSM or a support form. Do not present a capability recommendation that violates a hard constraint.1617## Step 1: region gate1819| Region | Rule |20| --- | --- |21| US / default | Full model catalog, subject to compliance gates. |22| EU-isolated | Restricted list. Preview Gemini models and several others are hidden. |23| India-isolated | Heavily restricted: only `custom-llm`, `glm-45-air-fp8`, `gpt-4o`, `qwen3-30b-a3b`, `qwen36-35b-a3b`, `speech-engine`. |2425Restriction is by deployment topology, not a toggle. Never offer to enable a region-hidden model.2627## Step 2: compliance gate2829Only relevant if the workspace has the corresponding entitlement. You cannot see workspace flags, so never assert an entitlement is active; the safe line is "your CSM can confirm this for your workspace." Never offer to enable HIPAA, PCI, or ZRM; they require contract changes.3031- HIPAA requires all three: enterprise tier with `force_logging_disabled=true`, an LLM in the HIPAA-compliant set, and a signed BAA (a contract artifact you cannot verify; route BAA questions to sales). Approved families include Claude 3.5/3.7 Sonnet, Claude 3 Haiku, Claude Haiku 4.5, Claude Opus 4.7, Claude Sonnet 4 / 4.5 / 4.6, `custom-llm`, Gemini 1.5/2.0/2.5 flash and pro variants, `gemini-3.1-flash-lite`, and `speech-engine`. Models outside the list are not HIPAA-eligible regardless of tier; preview/GA-pending Gemini models are excluded.32- PCI is deny-by-default when on (`pci_compliance_required`). Only pre-approved telephony providers, webhook domains, MCP URLs, and integration IDs work. Adding an integration requires CSM review; offer a support form, do not promise self-serve.33- ZRM disables audio/transcript storage; effective only when `force_logging_disabled=true`. Custom LLM plus ZRM additionally needs `is_convai_custom_llm_with_zrm_allowed`. No Stripe SKU; part of the enterprise contract.3435## Step 3: capability fit3637Recommend within the surviving candidate set. Weigh three axes against the agent's job.3839- Latency matters most for voice. In an STT to LLM to TTS loop, LLM time-to-first-token sits on the critical path and is the dominant lever on perceived delay. For real-time agents, bias toward the fastest tier that clears the quality bar.40 - Lowest latency: `gemini-3.1-flash-lite` / `nano` / small `qwen` / `*-mini` tiers and `speech-engine`.41 - Balanced: `gemini-3.5-flash`, `gpt-4o`, `gpt-5-mini`, `claude-haiku-4-5`.42 - Highest capability, higher latency: `claude-sonnet-4-6`, `claude-opus-4-7`, `gpt-5.x`, `gemini-3.x-pro` (where region/compliance permits).43- Intelligence. Reserve the top tier for genuinely hard reasoning, complex multi-tool orchestration, or nuanced instruction-following. Most transactional voice agents (booking, triage, FAQ, routing) run well on a mid tier and feel snappier. Over-provisioning intelligence usually costs latency the user will notice for quality they will not.44- Cost. Per-token price scales steeply with capability tier. Match tier to task; do not put a frontier model behind a deterministic IVR flow. For high-concurrency deployments, latency and cost compound.4546### Quick heuristic4748| Agent type | Priority | Typical pick (region/compliance permitting) |49| --- | --- | --- |50| Real-time phone / high concurrency | Latency, cost | `gemini-3.1-flash-lite` (ultra-low latency) or `gemini-3.5-flash` |51| Complex reasoning / multi-tool | Intelligence | `claude-sonnet-4-6`, `gpt-5.x`, `claude-opus-4-7` |52| HIPAA voice agent | Compliance, then latency | `claude-haiku-4-5`, `gemini-3.5-flash` (both in approved list) |53| India deployment | Region first | `gpt-4o` or `qwen36-35b-a3b` (catalog is tiny) |54| Custom/self-hosted model | Control | `custom-llm` (check ZRM + region eligibility) |5556## Apply the choice5758Set the model with `PATCH /v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID` and body `{"conversation_config":{"agent":{"llm":"<model-id>"}}}`. This is branch-scoped and ships on merge.5960## Output discipline6162- State the binding constraint explicitly ("In the EU deployment, X is not available, so among allowed models...").63- Give one primary recommendation plus a fallback, with the reason (latency vs intelligence vs cost).64- For anything contract-gated, end with the route: CSM for entitlement confirmation, a support form for new integrations/documents, and https://compliance.elevenlabs.io/ for attestations and signed DPA/BAA/sub-processor requests.65- Never claim an entitlement is active, never offer to enable a contract feature or a region-restricted model.
ElevenLabs