Coval Resources

Coval Platform Resource Overview

Reference for $ARGUMENTS. Use this to understand Coval's resource model before making API calls or exploring data.

Resource Hierarchy

All resources are scoped to your organization (determined by your API key).

Agent (22-char ID)
└── Mutation (26-char ID)
Test Set (8-char ID)
└── Test Case (22-char ID)
Persona (22-char ID)
Metric (22-char ID)
Run (22-char ID)
└── Simulation (22-char ID)
    └── Metric Output (26-char ID)
Run Template (22-char ID)
└── Scheduled Run (22-char ID)

Resources

Agent

An AI system being evaluated. Represents the customer's agent configuration.

• ID format: 22-char string
• Key fields: customer_agent_id (unique per org), display_name, model_type, phone_number, endpoint, metadata, prompt, language, test_set_ids, metric_ids, attributes
• Model types: MODEL_TYPE_VOICE, MODEL_TYPE_OUTBOUND_VOICE, MODEL_TYPE_CHAT, MODEL_TYPE_WEBSOCKET, MODEL_TYPE_API, MODEL_TYPE_ENDPOINT
• Connection types: Inbound voice, outbound voice, OpenAI endpoint, Pipecat, LiveKit, WebSocket
• API: GET/POST /v1/agents, GET/PATCH/DELETE /v1/agents/{agent_id}
• CLI: coval agents list|get|create|update

Mutation

A named configuration variant of an agent for A/B testing. Config overrides are deep-merged with the base agent's metadata at runtime.

• ID format: 26-char string
• Key fields: display_name, description, config_overrides (deep-merged JSON), parameter_values (flattened for display), status (ACTIVE/DELETED)
• Relationship: Belongs to an Agent via agent_id
• Unique constraint: One active mutation per (agent_id, display_name)
• API: GET/POST /v1/agents/{agent_id}/mutations, GET/PATCH/DELETE /v1/agents/{agent_id}/mutations/{mutation_id}
• CLI: coval mutations list|get

Test Set

A collection of test cases that define WHAT to test (scenarios, expected behaviors).

• ID format: 8-char string
• Key fields: display_name, slug, description, test_set_type, parameters (JSON template variables)
• Test set types: DEFAULT, SCENARIO, TRANSCRIPT, AUDIO, IVR, SCRIPT
• API: GET/POST /v1/test-sets, GET/PATCH/DELETE /v1/test-sets/{test_set_id}
• CLI: coval test-sets list|get|create|update

Test Case

An individual test scenario within a test set.

• ID format: 22-char string
• Key fields: input_str (the scenario/prompt), input_type, expected_behaviors (array of strings), expected_output_json, description, simulation_metadata_input
• Input types: SCENARIO (natural language task), TRANSCRIPT (OpenAI format conversation), AUDIO (pre-recorded file), SCRIPT (ordered lines), IVR, MANUAL
• Relationship: Belongs to a Test Set
• API: GET/POST /v1/test-cases, GET/PATCH/DELETE /v1/test-cases/{test_case_id}
• CLI: coval test-cases list|get|create|update

Persona

Defines HOW the simulated user behaves during testing (voice, personality, interruption style).

• ID format: 22-char string
• Key fields: display_name, simulated_user_prompt, voice_name, language_code, avatar_url, metadata
• Persona controls: Voice selection, background noise, interruption rate (NONE/LOW/MEDIUM/HIGH), silent mode, caller phone number
• Relationship: Referenced by Runs and Run Templates
• API: GET/POST /v1/personas, GET/PATCH/DELETE /v1/personas/{persona_id}
• CLI: coval personas list|get|create|update

Metric

Definition of an evaluation criterion and how to score it.

• ID format: 22-char string
• Key fields: metric_name (globally unique slug), display_name, description, metric_metadata (JSON config), output_type, category
• Output types: FLOAT, STRING, SET, BOOLEAN
• Features: Built-in metrics, custom LLM-based prompting, metric chaining, human review integration, expected-behavior evaluation
• API: GET/POST /v1/metrics, GET/PATCH/DELETE /v1/metrics/{metric_id}
• CLI: coval metrics list|get|create|update

Run

Top-level execution entity. Launches simulations of test cases against an agent.

• ID format: 22-char string
• Key fields: agent_id, persona_id, test_set_id, display_name, status, config (JSON), is_monitoring, tags, customer_metadata, scheduled_run_id
• Statuses: PENDING, IN QUEUE, IN PROGRESS, COMPLETED, FAILED, CANCELLED, DELETED
• Run types: Simulation run (is_monitoring=false) vs monitoring/conversation run (is_monitoring=true)
• Config options: iteration_count (1-50), concurrency (1-100), sub_sample_size, mutation_ids
• Total simulations = test_cases x iterations x (1 + mutation_count)
• API: GET/POST /v1/runs, GET/DELETE /v1/runs/{run_id}
• CLI: coval runs list|get|launch|watch|delete

Simulation

Result of running a single test case against an agent (one per test_case x iteration x variant).

• ID format: 22-char string
• Key fields: run_id, test_case_id, iteration, mutation_id, mutation_name, transcript, status, audio_length_seconds, external_conversation_id, tool_calls
• Mutation tracking: mutation_id and mutation_name identify which Mutation variant was used. Both are null for base agent simulations (no mutation).
• API: GET /v1/simulations, GET/DELETE /v1/simulations/{simulation_id}, GET /v1/simulations/{simulation_id}/audio
• CLI: coval simulations list|get|audio|delete
• Filtering: Supports mutation_id, mutation_name, agent_id, run_id, status, test_case_id, external_conversation_id, create_time

Metric Output

Result of evaluating a single simulation with a specific metric.

• ID format: 26-char string
• Key fields: metric_output_id, metric_id, value (float, string, or array), status
• Relationship: Child of Simulation, references a Metric
• API: GET /v1/simulations/{simulation_id}/metrics, GET /v1/simulations/{simulation_id}/metrics/{metric_output_id}

Run Template

Reusable, saved run configuration for scheduling.

• ID format: 22-char string
• Key fields: display_name, agent_id, persona_id, test_set_id, metric_ids, mutation_ids, run_config (JSON source of truth)
• Soft deletion: status=ACTIVE or DELETED
• API: GET/POST /v1/run_templates, GET/PATCH/DELETE /v1/run_templates/{id}

Scheduled Run

Cron/rate-based schedule that triggers runs from a template.

• ID format: 22-char string
• Key fields: template_id, schedule_expression (cron or rate), timezone, enabled
• Examples: rate(15 minutes), cron(0 2 * * ? *)
• API: GET/POST /v1/scheduled_runs, GET/PATCH/DELETE /v1/scheduled_runs/{id}

ID Formats Quick Reference

Common Workflows

Launch an evaluation

1. Identify Agent, Test Set, Persona (list with CLI or API)
2. POST /v1/runs with agent_id, test_set_id, persona_id, optional mutation_ids
3. Poll GET /v1/runs/{run_id} or use coval runs watch {run_id}
4. Results: GET /v1/simulations?filter=run_id="{run_id}" then fetch metrics per simulation

Compare mutations (A/B test)

1. Create mutations: POST /v1/agents/{agent_id}/mutations with config_overrides
2. Launch run with mutation_ids array
3. List simulations: GET /v1/simulations?filter=run_id="{run_id}"
4. Group by mutation_id (null = base agent)
5. Look up mutation names: GET /v1/agents/{agent_id}/mutations
6. Compare metrics across groups

Schedule recurring evaluations

1. Create Run Template with agent, persona, test set, metrics, mutations
2. Create Scheduled Run referencing the template with a cron/rate expression
3. Runs auto-launch and reference back via scheduled_run_id

Submit live conversations (monitoring)

1. POST /v1/conversations:submit with conversation data
2. Creates a Run with is_monitoring=true and single Simulation
3. Metrics evaluated in real-time

API Conventions

• Base URL: https://api.coval.dev/v1
• Auth: X-API-Key header
• Pagination: page_size (1-1000, default 50), page_token, response includes next_page_token
• Filtering: AIP-160 syntax — filter=status="COMPLETED" AND agent_id="abc123"
• Sorting: order_by=-create_time (prefix - for descending)

Tools & References

• OpenAPI Spec: GET https://api.coval.dev/v1/openapi — always fetch the latest spec before building integrations
• CLI: https://github.com/coval-ai/cli — install with brew install coval-ai/tap/coval
• MCP Server: https://github.com/coval-ai/mcp-server — Model Context Protocol server for LLM tool access
• Docs: https://docs.coval.dev