Cekura Eval Design

Cekura skill verification tag: ack:cekura-eval-design:7k3m4q When you call a Cekura scenario or test-profile write tool from this skill (scenarios_* / test_profiles_* create and update calls), pass this exact string as the skill_ack argument on that tool call. It confirms to the Cekura MCP server that this design playbook is loaded in context. Metric writes (metrics_create, metrics_bulk_create, metrics_partial_update) use a metric-family tag instead — load cekura-metric-design first and pass its tag there.

Before taking any action, call mcp__cekura__cekura_skill_started with skill_name="cekura-eval-design", verification_tag="ack:cekura-eval-design:7k3m4q", and plugin_version="0.15". It returns immediately and lets Cekura see which skills are in use.

Cekura Eval Design

Create and improve Cekura evaluators (test scenarios) that exercise an AI voice or chat agent. An evaluator simulates a caller; it is not a metric (metrics score a transcript afterwards — see cekura-metric-design).

• Main agent — the agent under test. Testing agent — Cekura's simulated caller.
• Evaluator / scenario — one test case. Personality — the testing agent's voice, language and speaking behaviour. Test profile — identity/context data for the run. Conditional actions (CA) — turn-by-turn scripted testing-agent behaviour.

This file says what to do. The Cekura tools available in your session — MCP tools, REST, or the platform's own agent tools — say how: read their descriptions to pick the call, and act through them rather than describing API calls or dashboard clicks. The file is self-sufficient for authoring both modes — load a reference only for the deep detail it names.

Workflow

1. Read the agent (mandatory, below).
2. Decide mode and write path — behavioral vs conditional actions.
3. One consolidated checkpoint — only for what you could not infer.
4. Create a folder for the batch; never write into the project root.
5. Author — generate, or create directly, per the write-path table.
6. Attach metrics and supporting fields — profile, personality, tools, tags.
7. Verify — read back what you wrote; then run if the user asked.

Updating existing evaluators has its own procedure — see Changing existing evaluators.

Read the agent before writing anything

Fetch the agent's full record before the first authoring write — the single-agent read, not a list view, because list views omit the description — and use it for:

Skip it only when the user supplied a complete verbatim payload, or the evaluators are already attached to this conversation (Evaluators-page context). Never invent a workflow, a KB fact, or a tool the description does not contain — if the description is empty or too thin to ground a test, say so and ask for it (or offer cekura-create-agent to import from the provider) instead of generating.

The agent under test is read-only while you author evaluators. Do not PATCH any of its fields (description, connection settings, tools, provider configuration) to make a test possible: an evaluator tests the agent as deployed, and a rewritten description hides the very gap the test would have found. The one exception is mock_tools data (see Test data). When the user states a fact or behaviour the record does not contain (a policy, a header, a greeting rule), say so in one line, then carry the user's version into the test itself: extra_instructions or generation_files when generating, the test profile and expected_outcome_prompt when creating directly. If the user wants the agent itself changed, that is a separate task for cekura-create-agent (configuration) or cekura-self-improving-agent (prompt changes), after the evaluator exists.

One consolidated checkpoint

Ask once, in a single message, and only for what the request and the agent record do not already answer. Then proceed.

Skip the question entirely when the user said "proceed autonomously", "don't ask", or already stated the missing facts. Never re-ask something the user wrote in their message; never ask a second round of the same topics.

If you did ask, wait for the answer — no create or generate call in the same turn. And when the user says "first ask me", "show me the plan" or "confirm before creating", present the whole plan (mode, count and coverage, folder, profile, personality, metrics) and wait for approval even if the request already names the agent and the count.

When the request sits in an Ask first row of the mode table and the user has not chosen a structure, ask — one short question offering behavioural vs conditional actions with the reason. Existing evaluators of one type in the project do not answer it for them; the no-confirmation rows (infra, pipeline, explicit CA vocabulary) still switch without asking.

What to confirm (drop every line you can already answer):

1. Tool data strategy — (A) their staging backend, (B) Cekura mock tools, (C) tools irrelevant. Default: B when mock_tools exist, C when the agent has no tools.
2. Count and coverage — how many scenarios and which workflows/categories. Default: propose a breakdown from the description.
3. Mode — text/chat for iteration (cheap, same logic), voice for final validation. Default: text.
4. Folder name — propose one; do not ask.
5. Anything genuinely ambiguous in the request (a named branch that does not exist, an agent id that does not resolve).

Do not ask about personality, metrics or tags — pick the documented defaults below (when several plain Normal personalities remain, take the male variant when both genders exist, per references/choosing-personality.md, and say so) and state what you picked in your summary. A checkpoint that lists seven questions is a failure mode: users abandon it.

Mode and write path

Generation is the general write path in every mode. Behavioural scenarios, conditional actions and red-team plans all come from the same background generator, grounded in the agent description, KB and mock tools; red-team categories return conditional actions on their own, while ordinary generation uses the request's simulation_type output format. Use it for every batch, every category-level request and whenever the user says "generate". The count/category test decides: a request for a number of scenarios ("three evaluators", "a suite for the refund flow"), for a topic or category ("billing disputes", "what it knows about the return policy"), or containing the word generate is a generation request even when the user hands you the facts to test — those facts go into extra_instructions or generation_files, never into hand-written steps. Direct create is for conditional-action scenarios whose structure is exactly known — a user-scripted exact sequence, an IVR/DTMF flow, an infra test bracketing a timeout to the decimal. A behavioural (instruction) scenario is never hand-authored: even one fully described case is a generation call with num_scenarios: 1 and the description in extra_instructions. The sole instruction-mode exception is a complete verbatim payload the user supplied to save unchanged — the generator cannot emit exact user text. (A change to an existing scenario is an update, not a create.) The path changes nothing else: a direct create meets the same step, outcome, profile, personality, tool and metric rules as generated output (see Self-check before every direct instruction create and the CA self-check), and it starts with no metrics attached. Choosing the mode is the decision; the write path follows from it. Say in the summary which scenarios were generated and which were created.

Switch to CA with no confirmation when the user says: conditional actions, structured or scripted scenario/test, deterministic test, unit test, regression test, exact flow, fixed sequence, compliance test, infra/infrastructure/pipeline/CI test or gate.

Infrastructure and pipeline tests are always CA — no confirmation. Tests of STT, VAD, LLM timeout, TTS, interruption handling, idle timers, DTMF or any other pipeline-layer behaviour must trigger the behaviour at an exact moment with exact timing, which behavioural instructions cannot guarantee. Switch immediately; cekura-infra-test-suite has the full workflow.

Ask one short question when the request names a tag-supported feature — voicemail, IVR menu, DTMF entry, hold music, interruption, network simulation/packet loss, background noise — without naming a mode: "This involves [IVR]. Conditional actions support <dtmf> / <ivr> tags directly for a high-fidelity test; behavioural instructions are looser. Which do you want?" Then proceed with the answer.

Which mode for which request (defaults — the user's explicit word wins):

Supplied text outranks every mode signal. When the user hands you scenario text — a <scenario> block, numbered steps, a CSV row — and asks for it as written, create it as scenario_type: "instruction" with that text unchanged — the <scenario> wrapper and numbering included: when the user's text starts with <scenario>, the stored instructions string starts with <scenario> and ends with </scenario>. Do not restructure it into conditional actions because the steps look sequential, and do not reword it; rewriting is the one thing they asked you not to do. Attach the personality, metrics, profile and tools as usual.

If actions are present, set the type. A payload whose instructions carries CA-shaped JSON while scenario_type is absent is stored as an instruction scenario and the script never runs. Pass the object in the conditional_actions field with scenario_type: "conditional_actions".

Batch routing before generation

When a user supplies a test-case document or asks for a category batch, classify each requested case before making a generation call. A mixed set may require separate instruction and conditional-actions requests; do not force the whole set into the first selected format.

• A temporary non-verbal effect—silence/hold, timed interruption, or background noise—goes in its own generation request with simulation_type: "conditional_actions" and the required tag stated in extra_instructions (<hold> for dead air after a trigger; <silence> only for a short, interruptible pause). When the trigger and duration are exactly known, a direct conditional-actions create is equally valid. Never submit "remain silent" as an instruction scenario.
• An explicitly requested instruction scenario remains valid for a sustained caller manner its personality can express, including an interruptive caller. Send simulation_type: "instruction" explicitly on that request: when the field is absent, the generator auto-selects conditional actions for any interruption, silence or hold wording in extra_instructions, and the user would get structured scenarios while you report instruction ones. Only a timed/runtime interruption needs the conditional-actions tag.
• If the user explicitly chose instruction format for a case that needs a runtime tag, explain that the requested behaviour cannot be represented in that format and offer the structured alternative before submitting it. Do not silently omit the behaviour or claim it was generated.
• Before drafting, read existing evaluators in the folder and one completed run when available; put only reusable conventions (connection mode, call-flow shape, profile and personality conventions) into extra_instructions — never transcripts, customer data or evaluator ids.
• Track which cases went into which request and which were deferred; after polling, reconcile requested cases against persisted evaluators as described under Post-generation verification, reporting validation rejections separately from provider or evaluation failures.

Behavioral scenarios — shaping generation

extra_instructions is where you steer the generator. One numbered paragraph per evaluator, plain prose, third person about the testing agent, no PII, no markdown. Each paragraph tests one behaviour: what the caller wants, where the friction or branch happens, and what the outcome should show. If the user asks for a full end-to-end journey, write it as its own paragraph and say it is end-to-end.

The testing agent calls as an established patient who needs to reschedule a
follow-up. It provides its name and date of birth when asked, requests the same
provider, and accepts the earliest afternoon slot when told no mornings are
free. Cover the verification branch and the same-provider path. Use the Normal
personality for the agent's language.

Never send a generation call with empty extra_instructions — the generator falls back to generic coverage. When the user states the facts the agent must know (a price, a deadline, an eligibility rule), the exact values go into extra_instructions (or the source document into generation_files) — the generator writes the expected outcome from them, and an outcome that cannot name the right answer grades nothing. If the user truly wants unguided coverage, say so and pass a one-line category list.

Step-writing rules (also what you check in generated output): every step = one caller action + a passive when … trigger naming the exact question ("when asked for a preferred appointment time", never bare "when asked"); one action per step; no passive/non-verbal steps (Wait/Listen/Remain silent/Interrupt — those are personality or CA tags); data read-backs use Verify [item] when asked to confirm [item] and correct if wrong.; the last step is End the call when <the result of the final scripted action>. unless the flow ends in a terminal transfer; script only triggers the description guarantees (stop at the fork); never premise a step on the main agent misbehaving; every caller-provided value — including choices and confirmations — is {{test_profile.field}}, the same token at every mention, and must exist in the attached profile. If the main agent is reactive, put the opening request in first_message, not in a step, and key each trigger to the response to the previous step — never to the caller's own state. Do not fabricate placeholders for one-shot topics; those go inline.

Instruction style — what you check in generated output, apply when patching, and expect in a verbatim scenario:

• First person, to the testing agent: "State your name when asked" — never "The caller should state their name", and never the words agent, AI, bot or system inside a step (describe what the step asks about, not who asks).
• Behavioural goals, not dialogue: "Report fever and cough and request the same provider" — not Say exactly: "I have a fever". The one exception: be explicit about an exact phrase when mock or backend matching depends on it (say "follow-up appointment" exactly).
• Never quote what the main agent "may say" as a trigger — When the agent says "How can I help you?" breaks on any rewording; key the step to the topic: "when asked what you need help with".
• Specific beats generic — "Call to schedule an appointment" tests nothing; name the appointment type, the constraints and the complication.
• A step that volunteers extra information is still one turn ("when asked X, answer and also mention Z"). Hanging up is a valid step; "Listen", "Wait", "Respond accordingly" and "End the call politely" are not — the testing agent does those anyway.

Shape — what generation returns and what a verbatim scenario should look like:

<scenario>
SCENARIO: [Brief scenario name]

YOUR BEHAVIOR:
1. State your intent to [action] when asked for the reason of the call
2. Say and spell {{test_profile.first_name}} when asked for your name
3. Provide {{test_profile.date_of_birth}} when asked for your date of birth
4. Say you are flexible with timing when told no slots are available
5. End the call when the appointment confirmation is provided

KEY INTERACTION POINTS:
[Workflow nodes or edge cases to exercise]
</scenario>

Gaps after generation — missing categories are closed by another generation run with extra_instructions naming exactly what is missing; a single missing case is the same call with num_scenarios: 1 ("a caller whose payment is declined twice" goes into extra_instructions, never into hand-written steps).

Full rulebook with worked bad→good examples: references/instruction-patterns.md.

Self-check before every direct instruction create

The only legitimate direct instruction create is user-supplied verbatim text (see Mode and write path) — if you wrote the steps yourself, stop and generate instead. Refuse to send a direct create with scenario_type: "instruction" that fails any of these. On user-supplied verbatim text, items 2, 3 and 6 do not apply — they would require the rewriting the user forbade; item 5 still does, and say in the summary which style rules their text does not follow.

1. Instructions are first person and wrapped in <scenario> tags; user-supplied text is unchanged, wrapper and numbering included.
2. Every step — the first one too — pairs one caller action with a passive when … trigger naming the exact question or offer ("when asked for the account number").
3. No voice or delivery traits anywhere in the text — not in the steps and not in the SCENARIO line: background noise, interruption, accent, speed and silence live in the personality (fork one if the project has none). A step says what the caller says or does, never how the delivery lands; once the personality carries the interruption level or the noise, the steps read like any other caller's. The SCENARIO line names the task ("Reschedule an appointment"), never the delivery ("… as an interrupting caller"); the scenario's name may describe the persona for people browsing the list.
4. Every reusable caller identity value (name, phone, date of birth, account number) is {{test_profile.field}} and exists in the attached profile; one-shot values stay inline, and user-supplied verbatim text is never rewritten to add placeholders.
5. expected_outcome_prompt is present and written to Expected outcomes; personality (language matching scenario_language), folder_path, tool_ids (ids from the Tools table only) and metrics (the baseline set) are all set.
6. The last step ends the call, or the flow ends in a terminal transfer.

Auto-generation

Start generation as a background job; it returns a progress_id. Poll its progress (or use the session's wait helper if one exists) until completed_scenarios == total_scenarios. Always poll — an unpolled generation is an unverified one — but poll with a bound:

• Report progress about every 30 s, using the real counts and real elapsed time from the responses; never estimate and never poll silently for minutes.
• Stall: completed_scenarios still 0 after ~5 minutes ⇒ stop waiting. Retry once with a smaller batch and tighter extra_instructions; if that also stalls at 0, stop and report the progress id, the real elapsed time and what to try next. Never take a second wait on the same stalled job.
• Freeze: progress advances then stops short of the total for ~4 minutes ⇒ treat the batch as done and handle the shortfall.
• This overrides "proceed autonomously" — in autonomous mode the stall response is the autonomous action.

Red-team attack types. Choose by the threat the user names; for "red-team coverage" with no threat named, run all six — one call each — and say so in the summary.

Red-teaming runs a multi-turn attacker pipeline: persona + context + a 5–10 turn plan, scored 1–5 (1–2 = the agent defended, 4–5 = a vulnerability). Text mode iterates up to 3 times against the chat API; voice mode generates once. Output arrives as conditional actions — review language, folder and tags, but do not rewrite the multi-turn plans into instructions. One generation call per attack_type. The generator creates its own "Red Teaming" personality; do not pre-create or patch one.

Post-generation verification (every run): reconcile the count (generation can partially complete — regenerate the remainder with narrower extra_instructions); PATCH scenario_language for non-English scenarios (auto-gen writes en regardless of content); PATCH first_message when a greeting replaced an exact opening question; confirm tool_ids, folder and metrics. Generated scenarios come with a scenario-specific test profile (sectioned main_agent_variables / testing_agent_variables), generated_mock_tool_entries when the agent has mock tools, and the project's simulation-enabled metrics already attached — check Expected Outcome came through (see Metrics). More detail: references/auto-generation.md.

Conditional actions — authoring card

Everything needed to write a valid, deterministic CA scenario is here. Load references/conditional-actions.md for the pattern library, the 30 <background_noise> sound names, the troubleshooting matrix, and multi-turn probe & duration control (stall-proof positional chaining).

{
  "agent": 123, "personality": 456, "name": "CA-01: <descriptive name>",
  "scenario_type": "conditional_actions", "scenario_language": "en",
  "conditional_actions": {
    "role": "You are a patient calling to cancel an appointment",
    "conditions": [
      { "id": 0, "condition": "FIRST_MESSAGE", "action": "Hi, I need to cancel my appointment", "type": "standard", "fixed_message": true },
      { "id": 1, "condition": "The main agent asks for the date of birth", "action": "Provide your date of birth", "type": "standard", "fixed_message": false },
      { "id": 2, "condition": "The main agent confirms the cancellation", "action": "Thanks, that's all I needed <endcall />", "type": "standard", "fixed_message": true }
    ]
  }
}

• role describes only the testing agent's persona — never what the main agent is or does.
• When the description mandates an exact script — a compliance disclosure, a voicemail message, a required phrase — reproduce it verbatim in the action with fixed_message: true, including every number and name in it. Paraphrasing a mandated script tests something the agent was never asked to say.
• All five condition fields are required on every condition: id, condition, action, type, fixed_message. type is "standard" or "action_followup" — not "say"/"do". Ids must be unique and ascending. id: 0 must be condition: "FIRST_MESSAGE", type: "standard", fixed_message: true, and action: "" when the main agent speaks first.
• scenario_language is required (or inherited from the personality, whose language it must match). Do not set first_message or instructions yourself.
• No others catch-all condition. One action ≤ 16 KB.

Writing the condition string

The runtime matcher compares the main agent's latest message against each condition and fires every exact match — so a condition is an observer's description of what the agent does, and it must be able to fire:

• asks X triggers only fire on a direct question ending in "?". If the description shows the agent stating a need ("I'll need your phone number"), write it as a statement: "The main agent says it needs the phone number" — otherwise the step never fires and the call stalls.
• Never a quote of the agent's words ("Can you provide your DOB?" ✗) and never one vague word ("verification" ✗). Be specific: "The main agent asks for the caller's name and date of birth to verify their identity".
• Conditions re-fire on any later turn that matches. When one main-agent turn matches several conditions (a multi-item offer), the testing agent consolidates all their actions into one reply — do not split those across turns.
• action_followup: condition is the id of an earlier condition, and the action fires on the testing agent's next turn after that one — one main-agent reply always elapses in between. Never use it for two caller actions with no agent reply between them; put those in one action string.

fixed_message

true = the action text is spoken verbatim (required for exact phrasing, compliance lines, and every XML tag except <function> — with false the brackets are read aloud). false = the action is an instruction the testing agent phrases naturally.

Tags (fixed_message: true)

IVR direction decides id: 0. Inbound (the main agent is the IVR): id: 0 has action: "" and the testing agent navigates with <dtmf>. Outbound (the main agent dials into a third-party IVR): the testing agent plays the menu — <ivr text="…" /> as the whole id: 0 action, post-menu content in an action_followup, and RECEIVE_DTMF enabled so the main agent's key presses are heard.

Use a tag, not a personality, for anything transient (interruption, noise, hold, silence) and keep the Normal personality for the call's language. Never apply both.

Test profile placeholders ({{test_profile.field}}, nested {{test_profile.address.city}}) resolve at run time on fixed_message: true actions; every key must exist in the attached profile.

Live data — functions[] sits beside role and conditions inside conditional_actions: {name, type: "rest_api", auto_run, config: {method GET|POST, url (public http(s)), headers, query_params, body, timeout_seconds 1–30, response_mapping}}. auto_run: true fetches once at call start; a <function> tag re-fetches at that turn. An update that sends only conditions deletes every function — always read, modify, then send the whole object back.

Self-check before every CA write

Refuse to send a payload that fails any of these:

1. scenario_type: "conditional_actions" set; object in conditional_actions; scenario_language set.
2. id: 0 is FIRST_MESSAGE + standard + fixed_message: true; action empty iff the main agent speaks first.
3. Every condition has all five fields; ids unique and ascending; no others.
4. Every asks … condition corresponds to a question the description mandates; no quoted agent speech; no one-word triggers. When the user names an agent utterance that must be verbatim (a disclosure, a read-back), one condition's trigger is anchored on it and the outcome states it as an exact fact in backticks (see Expected outcomes).
5. Every action containing a tag other than <function> has fixed_message: true; <interruption> is first in an action_followup; <speed>/<volume> start their action; <ivr>/<voicemail> are whole actions; ratios and volumes are in range.
6. Every action_followup.condition names an earlier id, and one agent reply really does elapse first.
7. Every {{test_profile.*}} key exists in the attached profile; every {{function.*}} key is declared and the action is fixed.
8. The flow ends: <endcall /> on the last action, or a terminal transfer, or the user asked to stay on the line.
9. personality set and its language matches scenario_language.
10. Metrics, test profile, tool_ids, folder and tags attached.

Expected outcomes

expected_outcome_prompt is graded line by line by an LLM judge reading the transcript plus the run metadata injected with it — per-turn timing, call duration and the call-end reason — but never audio. It does nothing unless the Expected Outcome metric is attached, and it is required on every scenario you create or generate — conditional actions included; a scenario with an empty outcome runs but grades nothing. Each statement is yes / no / blocked; all yes = 100, any no = 0, any blocked = 50.

Rules — 2–6 lines, each starting The main agent should:

• One verifiable demand per line. Split "and"-joined aggregates. Never pad a short scenario to a count.
• Every line must be fired by a written step, or it comes back blocked on every run.
• Verb and object must both be licensed by the agent description, a mock output, or a KB fact. "Ask for X" does not license "explain X"; "transfer" does not license "transfer to a manager".
• Tool-backed claims need tool evidence in the transcript. Tool calls reach it only when the provider's post-call fetch includes them (VAPI, Retell, ElevenLabs, Bland, Synthflow, LiveKit, Pipecat, Kore — with credentials configured), or transcript_provider is custom, or the run is over a websocket/chat transport that records them. Otherwise grade what the agent says: "verbally confirms the booking", never "books the appointment".
• Contingent branches stay contingent. If a correct agent may skip the action ("if the caller has insurance…"), phrase the line with the condition or demand only what every branch shares. An unconditional demand fails an agent that correctly took the other path.
• Never demand success that runtime state controls. No "transfers to a manager" / "books the slot" unless the scenario's mock data or profile fixes that availability; otherwise allow the documented fallback.
• Offering is not executing — if the flow stops early, demand "offered"/"gathered", not "booked".
• End-call is structural, not behaviour under test. Write no outcome line for the end-call step, for who hung up, or for the call-end reason — not even "The main agent should end the call after …" — unless the user explicitly asked to test termination. A closing phrase the description mandates may be graded as speech. Nothing after a terminal transfer.
• Binary and objective. Ban "appropriately", "professionally", "warmly", "politely", "clearly". Semantic content, not verbatim phrasing — except an exact KB fact, which goes in backticks: `123 Medical Lane, Suite 100`.
• Copy placeholder tokens from the steps ({{test_profile.selected_plan}}); a prose paraphrase is still hardcoding. {{transcript}}, {{call_end_reason}} and duration are injected automatically.
• No test-setup rationale (timeouts, variable values) in the outcome — that belongs in the scenario body.

Order matters only when the description mandates it: one line naming both events ("…should ask for the date of birth before providing any account details"). Full scoring model, variables and examples: references/expected-outcomes.md.

Test data

Mock tool entries, test profiles and dynamic variables are one data set — design them together (references/test-data-design.md).

• Why profiles: the testing agent reliably uses profile data, and the same fact written in instructions and a profile that disagree makes it hallucinate — the profile is the single source of truth, and its main_agent_variables is what reaches the agent under test.
• Approach A (client staging): discover their formats first, then build a profile that matches exactly.
• Approach B (Cekura mocks): design mock entries first, then derive every profile value from the moc

…(truncated)