# Cekura Metric Design

> Use when the user asks to "create a metric", "write a metric", "design a metric", "build a metric for", "evaluate agent performance", "measure call quality", "track a KPI", "add a workflow metric", "improve my metric", "fix a metric", "debug metric results", "set up quality scoring", or "what metrics do I need". Also relevant when discussing LLM judge prompts, custom code metrics, evaluation triggers, VALID_SKIP patterns, section extraction, or metric best practices for Cekura voice AI agents. Covers both creating new metrics and reviewing, iterating on, or troubleshooting existing ones.

- Skill: `cekura-ai/cekura-metric-design` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add cekura-ai/cekura-metric-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/cekura-ai/cekura-metric-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: cekura-ai (https://skillmd.com/u/cekura-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/cekura-ai/cekura-metric-design

---


<!-- cekura-ack-tag: ack:cekura-metric-design:6n2q5r -->
> **Cekura skill verification tag:** `ack:cekura-metric-design:6n2q5r`
> When you call a Cekura metric write tool from this skill (`metrics_create`, `metrics_bulk_create`, `metrics_partial_update`), pass this exact string as the `skill_ack` argument on that tool call. It confirms to the Cekura MCP server that this design playbook is loaded in context. Scenario / test-profile writes use an eval-family tag instead — load `cekura-eval-design` first and pass its tag there.

Before taking any action, call `mcp__cekura__cekura_skill_started` with `skill_name="cekura-metric-design"`, `verification_tag="ack:cekura-metric-design:6n2q5r"`, and `plugin_version="0.15"`. It returns immediately and lets Cekura see which skills are in use.

# Cekura Metric Design

## Purpose

Guide the creation of effective Cekura metrics that accurately evaluate AI voice agent call quality. Metrics measure call quality after the fact by evaluating transcripts against defined criteria. Each metric targets a specific workflow or KPI that needs tracking per call.

## Performing Platform Actions

When this skill suggests creating, listing, updating, or evaluating something on Cekura, **prefer using available platform tools over describing API calls or dashboard steps**. In Claude Code with the Cekura plugin installed, these tools are auto-configured and handle authentication, parameter validation, and error handling for you. Fall back to direct API endpoints or dashboard guidance only when no tools are available in the current session.

## Core Terminology

- **Main agent**: The client's AI voice agent being tested
- **Testing agent**: Cekura's simulated caller that exercises the main agent
- **Metric**: A post-call evaluation that scores a transcript
- **Evaluator/Scenario**: A test case that simulates a caller (separate concept — see cekura-eval-design skill)

## The Metric Creation Workflow

Follow this workflow every time. Skipping steps (especially step 2) leads to metrics that miss edge cases.

1. **Gather context** — Understand the client's use case, what they care about, get sample conversation IDs with expected outcomes
2. **Fetch real transcripts** — Pull 3-5 actual `transcript_json` records from the Cekura API. Study: what roles appear (`Main Agent`, `User`, `Function Call`, `Function Call Result`), what timestamps are available, how tool calls are structured, what the conversation flow looks like in practice. Metrics written without reading real data will miss edge cases.
3. **Identify the signal** — What specific thing in the transcript indicates pass vs fail? A tool call, a timestamp gap, a phrase, a behavioral pattern?
4. **Write the prompt** — Use proven structures (see below), grounded in what the real transcripts look like
5. **Deploy and test** — Create the metric via API, run on sample conversations, compare to expected outcomes
6. **Iterate** — Adjust the prompt based on results, re-run, repeat until the metric matches expectations on all samples. Plan for at least one iteration — the first run reveals measurement issues.

## Metric Types

### llm_judge (preferred default)

An LLM evaluates the prompt in the `description` field against the call transcript. **Prefer llm_judge over custom_code.** Custom_code seems appealing for "objective" checks (timestamps, tool call presence) but is brittle in practice. Voice AI transcripts have messy timing — agents transfer mid-tool-chain, background tasks complete after speech resumes, timestamps overlap. An LLM reading the transcript handles these nuances naturally. Express measurements in natural language, not code.

**Critical:** The evaluation prompt goes in the `description` field, NOT the `prompt` field.

### custom_code (Python on Lambda)

Python code in `custom_code` field. Has access to `data` dict, `evaluate_basic_metric()`, and upstream metric results. Reserve for cases that genuinely need programmatic logic.

**Use only when:**
- Gating on upstream metric results (`data.get("Exact Metric Name")`)
- Section extraction from agent description (Pythonic pattern)
- Multiple LLM calls with different prompts based on conditions
- N/A short-circuiting before calling the LLM

## Metric Evolution Path

Start as `llm_judge` for rapid iteration. Once the prompt is validated (through labs/feedback), convert to `custom_code` with section extraction for production. Cekura allows a metric to have both `description` (llm_judge prompt) AND `custom_code` — the active type is toggled. This means the LLM prompt can be refined through labs, then the custom_code version uses that same prompt with targeted context extraction.

## Eval Types

| Eval Type | Output | Use For |
|-----------|--------|---------|
| `binary` | TRUE/FALSE | Pass/fail checks — flow compliance, soft-skill/quality assessments |
| `enum` | String from defined values | Classification tasks |
| `numeric` | Float score | Scoring tasks |
| `continuous_qualitative` | Continuous score | Continuous quality assessment |

## LLM Judge Prompt Structure

Two proven structures exist. See `references/prompt-patterns.md` for full templates.

### Structure A: Sectioned (best for multi-criteria metrics)
1. **SCOPE & FOCUS** — What this metric evaluates ONLY + what to IGNORE
2. **DO NOT FLAG** — Common false positives: behavioral patterns that look like fails but aren't for THIS metric
3. **INPUTS** — Only relevant template variables
4. **SECTIONS** — Numbered evaluation criteria with pass/fail examples
5. **FAILURE CONDITIONS (Only These Count)** — Narrow, closed list of what constitutes a failure
6. **SAFEGUARDING NOTES** — Spirit vs letter overrides
7. **OUTPUT INSTRUCTIONS** — Return format, timestamps for failures

### Structure B: Narrative (best for behavioral/timing metrics)
1. **SCOPE & FOCUS** — What this metric evaluates ONLY + what to IGNORE
2. **DO NOT FLAG** — Common false positives specific to this metric
3. **CONTEXT** — What this call type looks like, why the metric matters
4. **WHAT TO LOOK FOR** — Specific items in the transcript (tool names, phrases, patterns)
5. **FAILURE CONDITIONS (Only These Count)** — Narrow, closed list of specific failure patterns
6. **NUANCES** — Edge cases, overrides, things that look like fails but aren't
7. **OUTPUT** — TRUE/FALSE/N/A with timestamps and evidence

Being explicit about PASS vs FAIL with real examples from actual conversations is the single most impactful thing for prompt quality. Vague criteria like "agent should be responsive" produce inconsistent results.

### Anti-Cross-Pollination Scoping (when using `{{agent.description}}`)

The most common source of false failures: a metric using `{{agent.description}}` fails based on rules from an unrelated flow (e.g., Emergency metric fails because of a Booking Flow rule).

**Three-layer scoping pattern**: SCOPE & FOCUS ("evaluates X ONLY"), DO NOT FLAG (enumerate false positives by behavioral pattern), FAILURE CONDITIONS (narrow closed list).

**See `references/advanced-patterns.md`** for full structure and the rule that all scoping language must be concept-based, never hardcoded to a specific agent's section names.

### Available Template Variables

| Variable | Description |
|----------|-------------|
| `{{transcript}}` | Full conversation text |
| `{{transcript_json}}` | Structured transcript with timestamps |
| `{{dynamic_variables}}` | Full blob of custom variables from calls |
| `{{dynamic_variables.keyName}}` | Specific dynamic variable by key (dot notation preferred) |
| `{{agent.description}}` | Main agent's system prompt |
| `{{metadata}}` | Call metadata |
| `{{call_end_reason}}` | How the call ended |

Include only variables relevant to the specific metric. Listing all variables creates noise and dilutes evaluation focus.

When using `{{metadata}}`, point to specific metadata fields the LLM judge should reference (e.g., "Check `metadata.appointment_id` to verify booking was created").

## The Spirit vs Letter Principle

**This is the most critical concept in metric design.**

Agent descriptions describe the intended functionality of the main agent, but must not be taken literally by the evaluator. Understand the **intent** behind each instruction and write the metric to capture the **spirit**, not the literal text.

**Example:** Agent description says "ask only 1 question at a time"
- **Spirit:** Prevent cognitive overload on the caller
- **Literal (wrong):** Fail any turn with more than one question mark
- **Correct metric behavior:**
  - PASS: "Are you the owner of 123 Easy St? Can I get your name?" (related data cluster)
  - PASS: "Is this a new issue, or an existing one?" (A/B rephrasing = single question)
  - FAIL: "Does Thursday work? Also, did you get our text message?" (unrelated questions)

When uncertain about the intent behind an instruction, **ask the user** to clarify before encoding it into the metric. Include explicit safeguarding examples in the prompt showing what should and should not be penalized.

## Trigger Design

| Trigger Type | When to Use |
|-------------|-------------|
| `"always"` | Metrics that apply to every call (soft skills, business context) |
| `"custom"` with `llm_judge` trigger | Conditional metrics (booking flow only fires when booking intent detected) |
| `"custom"` with `custom_code` trigger | Complex trigger logic requiring code |
| `"automatic"` | Let Cekura auto-determine (less control) |

Use the `generate_evaluation_trigger` endpoint (see `references/api-reference.md`) to auto-generate trigger prompts from metric descriptions. Triggers can be layered in specificity — e.g., one trigger fires on any onboarding call, another fires only when the user gets stuck.

### Two-Layer N/A Strategy

Triggers and metric descriptions handle N/A at different levels:

- **Trigger-level N/A (first defense):** The trigger gates out obviously irrelevant calls BEFORE the metric runs. This saves LLM cost. Example: a Booking Flow metric's trigger checks if booking intent exists — if not, the metric doesn't fire and outputs N/A.
- **Description-level N/A (nuanced cases):** The metric prompt itself handles edge cases that need transcript context to determine. Example: a call has booking intent (trigger fires) but the caller hangs up before the flow starts — the metric description returns N/A with "VALID_SKIP: caller disconnected before booking could begin."

Design triggers to catch the obvious non-applicable calls; design the metric prompt to handle the nuanced edge cases that require reading the transcript.

### Trigger Prompt Template

Write triggers with the positive-then-negative pattern:

```
Evaluate whether this call involves [specific scenario].

Return TRUE if ANY of these indicators are present:
- [Positive indicator 1]
- [Positive indicator 2]

Do NOT trigger if ANY of these apply:
- Call is under 30 seconds or contains no substantive interaction beyond a greeting
- Line disconnection / voicemail / outbound non-engagement
- [Specific exclusion for this metric — e.g., "Emergency-flow transfers (covered by Emergency metric)"]
- [Another exclusion]

Be inclusive — if there's reasonable evidence the scenario occurred, return TRUE.
```

**Always include the short-call exclusion.** Calls under ~30 seconds (hang-ups, wrong numbers, voicemails) produce false positives/negatives on every metric. Gate them out at the trigger level.

### Trigger Produces N/A Output

When `evaluation_trigger: "custom"` and the trigger returns false, the metric outputs N/A — it is not evaluated. This means even binary metrics (True/False) can have three outcomes: True, False, or N/A. This is correct behavior for conditional metrics.

## Key Patterns

### VALID_SKIP Pattern

For legitimate deviations where the metric should not apply (tool failures, user hangup before flow starts, caller requesting transfer immediately). The LLM returns explanation starting with "VALID_SKIP:" and the custom_code wrapper converts to `_result = None`.

### Gated Metrics

Access upstream metric results via `data.get("Exact Metric Name")`. The key must match the upstream metric's `name` field exactly. Use to branch evaluation logic based on prior classification.

### Pythonic Section Extraction

Extract only relevant sections from agent description before passing to LLM. Prevents context drift from irrelevant description sections and reduces token usage. See `references/pythonic-patterns.md` for the extraction utility.

### N/A Conditions

Check first for conditions where the metric should not apply:
- Immediate transfer/human request within first 1-2 exchanges
- Caller hangup before flow begins
- Out-of-scope caller (wrong number, sales call)
- Infrastructure failure preventing flow execution
- Agent description lacks the relevant section (for optional workflows)

### Dynamic Variable-Driven Generalized Metrics

For clients that inject per-call `dynamic_variables` (e.g., per-node system prompts, feature flags, employment types), create metrics that adapt to each call instead of hardcoding behavior. **Pattern: one metric per injected prompt variable.** Each metric references ONLY its specific `{{dynamic_variables.promptName}}`, not the full blob or `{{agent.description}}`.

**See `references/advanced-patterns.md`** for the example prompt structure and the discovery workflow for finding dynamic variables in real calls.

### Tool Call Hallucination Metrics

For agents with detailed tool definitions, build a metric that evaluates whether the agent called the **correct tool for each situation** — "action hallucination" (wrong action) vs "fact hallucination" (wrong information). Pattern: extract tool→scenario mapping from agent description, encode as explicit FAILURE CONDITIONS (closed list), DO NOT FLAG API errors / known quirks.

**See `references/advanced-patterns.md`** for the full prompt structure and TOOL-TO-SCENARIO MAPPING template.

## Baseline Metrics — Always Recommend

Every agent should have at minimum these predefined metrics enabled for both observability and simulations:

| Metric | Purpose | Why It Matters |
|--------|---------|----------------|
| **Expected Outcome** | Checks if the agent achieved the scenario's expected result | Without this, runs pass/fail based only on call completion — not correctness |
| **Infrastructure Issues** | Flags silent periods, connection drops, agent non-response | Catches issues like agent going silent for 10+ seconds that aren't visible in pass/fail |
| **Tool Call Success** | Monitors whether tool calls succeed or fail | Requires provider integration (assistant IDs + API keys) to get toolcall data in transcripts |
| **Latency** | Measures response time | Identifies performance degradation |

**Two-step activation required:** Metrics must be (1) toggled on for simulations at the project level AND (2) added to individual evaluators. Missing either step means metrics won't fire. Without metrics enabled, users get false passes and must manually review every run.

**Simulation cost scales with attachment.** Every metric attached to a scenario is scored on every run of it, and a project rubric rule over that metric gates the run's `success`. A flow-specific LLM metric that the project attaches to every scenario should carry a trigger that returns N/A where the flow does not occur, so unrelated scenarios are neither billed nor failed by it.

**Expected Outcome is transcript-only — it cannot evaluate audio-layer behavior.** Expected Outcome reads the conversation text to determine whether the agent achieved its goal. It has no visibility into silences, interruptions, barge-ins, audio dropouts, or other voice-channel signals. Do not rely on Expected Outcome to catch these. For anything that depends on the audio stream rather than conversation content, use predefined metrics instead.

**Toolcall data prerequisite:** Tool Call Success and advanced monitoring require the agent to have its provider assistant ID configured on Cekura and complete call data being sent. If transcripts are missing toolcall data, recommend the user configure their provider integration.

## Output Requirements

All metrics must require:
- Brief explanation of the result (what happened and why)
- For failures: specific timestamps in MM:SS format where violations occurred
- For metadata-based checks: reference the specific metadata fields examined

## Common Custom Metrics Worth Suggesting

Beyond the baseline predefined metrics, these are commonly valuable custom metrics based on patterns seen across clients:

- **Question stacking / information dumping** — Agent asking 3+ unrelated questions in one turn or dumping large blocks of information. Poor UX that overwhelms callers.
- **Workflow adherence** — Agent follows the defined flow steps in order (booking, verification, cancellation, etc.)
- **Soft skills** — Tone, empathy, appropriate greetings, not exposing system internals
- **Business context accuracy** — Agent provides correct business information (hours, addresses, pricing)
- **Transfer/callback handling** — Agent follows proper protocol when transferring or scheduling callbacks

## Operational Rules

### Cost Guard — Never Evaluate >100 Calls Without Confirmation

Each evaluation costs the client real money. Before evaluating metrics on a batch of calls, ALWAYS query the call count first (use `page_size=1` and read the response) and report the number to the user. If count > 100, stop and ask for explicit approval before proceeding. Use `page_size` parameter (up to 200) instead of paginating, and use server-side filters (`agent_id`, `project`, `timestamp__gte`/`timestamp__lte`) to scope calls.

### Manual Fix First, Then Labs

When metrics have systemic issues (high false-fail rates), do NOT jump straight to labs feedback. Instead:
1. Read failure explanations and categorize root causes (e.g., extra_questions, end_protocol, should_be_na)
2. Write manual prompt fixes targeting the dominant failure categories
3. PATCH the updated descriptions via API
4. Re-evaluate a sample of 20-30 calls per metric to validate the fixes
5. THEN use labs feedback for remaining edge cases that manual fixes didn't catch

This avoids wasting labs iterations on issues that are clearly fixable by prompt editing.

## Common Pitfalls

- **Writing metrics without reading real transcripts first** — always fetch and study actual transcript_json before writing
- Putting the prompt in `prompt` field instead of `description` for llm_judge
- Using deprecated types (`basic`, `custom_prompt`) — API returns 400
- Using `custom_code` for checks the LLM can handle naturally (timestamps, tool call detection)
- Not matching upstream metric name exactly for gated metrics
- Passing full agent description when only a section is relevant (context drift)
- Missing VALID_SKIP handling in custom_code metrics
- No N/A conditions for conditional metrics
- Taking agent description instructions literally instead of capturing their spirit
- Not including safeguarding examples for nuanced evaluation criteria
- Omitting timestamps in failure explanations

## Next Steps

After creating a metric, the user typically needs:
- **Validate it on real calls** → use the evaluate-calls flow (see `references/api-reference.md`)
- **Iterate on accuracy** → invoke **cekura-metric-improvement** to run the labs feedback cycle
- **Design test scenarios that exercise this metric** → invoke **cekura-eval-design**

## Documentation

- Public docs: https://docs.cekura.ai
- LLM-friendly docs: https://docs.cekura.ai/llms.txt
- Concepts: https://docs.cekura.ai/documentation/key-concepts/

See `references/api-reference.md` for complete endpoint documentation and field schemas.

## Additional Resources

### Reference Files (loaded on demand)

- **`references/prompt-patterns.md`** — Full LLM judge prompt templates with real examples
- **`references/pythonic-patterns.md`** — Section extraction utility and custom_code patterns
- **`references/advanced-patterns.md`** — Anti-cross-pollination scoping, dynamic-variable-driven metrics, tool-call hallucination metrics
- **`references/api-reference.md`** — Complete Cekura metrics API endpoints and schemas

### Example Files

- **`examples/llm-judge-metric.md`** — Complete llm_judge metric example (sectioned structure)
- **`examples/narrative-metric.md`** — Complete llm_judge metric example (narrative structure)
- **`examples/custom-code-metric.py`** — Complete custom_code metric with gating and VALID_SKIP
- **`examples/section-extraction-metric.py`** — Pythonic metric with agent description scoping