Simulations

Vapi Simulations

Build realistic conversation tests in five layers: a personality controls the AI tester, a scenario defines its intent and measurable outcomes, a simulation pairs them, a suite groups simulations, and a run executes them against an assistant or squad.

Source and Safety Rules

• Verify live payloads against the current Vapi documentation MCP, API reference, or public OpenAPI before sending them. Simulations use the /eval/simulation API family.
• Never print, request in chat, or embed API keys, provider secrets, credential values, private webhook URLs, or real customer data.
• Treat running a simulation as an external action. It can consume credits, use concurrency, send webhooks, and call the target's real tools unless they are mocked.
• Do not run, cancel, update, or delete resources unless the user clearly requests that operation. Draft configurations when mutation is not requested.
• Resolve every assistant, squad, personality, scenario, simulation, suite, tool, structured-output, and credential ID from user input or the API. Never invent an ID.
• Do not create legacy Test Suites. Use Evals for deterministic turn-by-turn checks and Simulations for dynamic conversations over chat or voice.

Procedure

1. Choose the test type and execution mode.

   • Use Simulations for multi-turn behavior, personality variation, squad handoffs, realistic tool paths, or audio behavior.
   • Use Evals instead when the requirement is an exact response, regex, fixed mock conversation, or precise tool-call argument check.
   • Return a test plan or payload when the user asks to design, draft, review, or explain. Perform live mutations only when explicitly requested and VAPI_API_KEY is available.

2. Inspect the target and existing test resources.

   • Fetch the assistant or squad and identify its core paths, guardrails, tools, variables, languages, and failure behavior.
   • List existing personalities, scenarios, simulations, suites, and reusable structured outputs before creating duplicates.
   • Reuse an existing resource only when its intent and configuration match unambiguously. Otherwise create a clearly named new resource or ask the user to choose among plausible matches.

3. Design coverage before payloads.

   • Start with one smoke simulation for the core path, one or two required Boolean outcomes, chat transport, and one iteration.
   • Add regression simulations for repaired defects. Add separate edge cases for ambiguity, interruption, refusal, unavailable dependencies, failed tools, escalation, and handoffs.
   • Keep scenario intent, personality behavior, and evaluation criteria independent so each can be reused.
   • Name resources by behavior and expected outcome, not implementation details.

4. Define the personality.

   • Prefer a suitable existing personality when available.
   • When creating one, provide a complete valid assistant configuration for the AI tester. Put stable temperament, speaking style, and caller behavior in its system prompt; put the situation-specific goal in the scenario.
   • Use the create-assistant skill to assemble or validate the personality's assistant configuration when available.
   • Configure voice and transcriber only when voice runs need them. Chat runs use the personality's model but skip its audio path.

5. Define the scenario and evaluations.

   • Write instructions as the AI tester's intent and facts. Describe the goal and constraints without scripting the target assistant's answer.
   • Make each evaluation measure one observable outcome. Prefer descriptive Boolean outputs for pass/fail facts and numeric outputs for thresholds.
   • Provide either structuredOutputId or inline structuredOutput, never both. Inline outputs require name and a JSON schema.
   • Match the expected value type to the evaluated primitive. Use = or != for Boolean and string; numeric types also support >, <, >=, and <=.
   • Keep important criteria required: true. Use optional criteria only for diagnostics that must not fail the simulation.
   • Object structured outputs may be evaluated through a primitive leaf using path. Do not compare an object or array directly.

6. Isolate side effects and runtime context.

   • Inspect the target's configured tools before every run. Mock any tool whose real execution could write data, contact people, spend money, or make the test non-deterministic.
   • Match each toolMocks[].toolName exactly. The mock result is always a string; encode JSON as a string when the target expects JSON-shaped output.
   • Assume every unmocked tool remains live in both chat and voice simulations.
   • Put test values for {{variables}} in targetOverrides.variableValues. Use synthetic data and keep secrets in Vapi credentials.
   • Configure simulation.run.started or simulation.run.ended hooks only when requested. Prefer server.credentialId to inline authorization headers.

7. Create and verify reusable resources.

   • Create in dependency order: personality and scenario, then simulation, then optional suite.
   • Require 201 for create operations. Verify returned IDs and the fields that define the test.
   • For updates, fetch the current resource first. Omit unrelated scalar fields and send the complete intended value for any array being changed; suite simulationIds and targetAssignments replace their existing arrays.
   • Re-fetch after update. Deleting a suite or other simulation resource is permanent; verify the exact ID and dependency impact first.

8. Run deliberately.

   • Prefer vapi.webchat for fast prompt, tool, and conversation-logic iteration.
   • Use vapi.websocket for speech recognition, voice output, interruptions, recordings, or final end-to-end validation.
   • Start with one iteration. Increase iterations only to measure behavioral consistency after a single run is valid.
   • Before sending the run, recap the target, simulations or suite, transport, iterations, tool mocks, and any remaining live side effects.
   • Create the run with POST /eval/simulation/run and require 201. Return the run ID and dashboard url when present.

9. Monitor and diagnose results.

   • Poll GET /eval/simulation/run/{id} until status is ended; do not treat queued or running as success.
   • Fetch GET /eval/simulation/run/{id}/item and inspect every item. A passing group has items to evaluate, zero failed or canceled items, and every required evaluation passes.
   • Report actual versus expected values, extraction errors, skipped evaluations, failure reasons, transcript evidence, transport, and iteration number.
   • Diagnose the failing layer before changing the assistant: target runtime failure, scenario ambiguity, personality behavior, tool mock mismatch, structured-output extraction, or genuine assistant behavior.
   • Keep the evaluation stable when fixing the assistant. Change expected criteria only when the business requirement changed.

10. Handle failures honestly.

• On 400, compare the request with the current schema and correct one unambiguous validation issue before at most one retry.
• On 401 or 403, stop for authentication or permission. On 404, report the missing dependency. On 409 or concurrency errors, inspect GET /eval/simulation/concurrency and active runs. On 5xx, report the service failure.
• Cancel only queued or running groups or items. Never claim a run, cancellation, mutation, or pass succeeded until the corresponding API response is verified.

API Implementation

Read Simulation API Reference before producing REST code, making a live request, configuring hooks or mocks, or interpreting run results. Use direct REST unless the current official Vapi SDK documentation explicitly exposes the required simulation resource and method; never invent SDK method names.

Output Contract

Return only the sections relevant to the request:

• Test strategy: target behavior, coverage, and why Simulation rather than Eval
• Resource plan: personality, scenario, evaluations, simulation, and suite
• Side-effect review: mocked tools, live tools, hooks, variables, transport, iterations, and expected cost/concurrency impact
• Save-ready JSON or implementation code
• Created resource IDs and verified fields, when mutations succeeded
• Run ID, dashboard URL, status, item counts, and per-evaluation evidence, when a run was requested
• Failure diagnosis and the smallest recommended next change

Public Sources

• Simulations overview
• Simulations quickstart
• Simulations advanced
• Manage simulations
• Vapi API reference index and OpenAPI