Frootai Frootai Evaluate Voice AI Agent

Evaluate Voice AI Agent

When to Use

• Evaluate intent detection accuracy across utterance variations
• Measure dialog completion rate (user goal → successful resolution)
• Validate latency budget (STT + LLM + TTS < 2s)
• Assess voice quality (naturalness, prosody appropriateness)
• Test multi-turn context retention over long conversations

Voice Agent Metrics & Targets

Step 1: Test Intent Detection

Create diverse utterance test set:

{"utterance": "Book a meeting for tomorrow at 3pm", "expected_intent": "schedule_meeting", "entities": {"date": "tomorrow", "time": "3pm"}}
{"utterance": "Can you set up a call with the team?", "expected_intent": "schedule_meeting"}
{"utterance": "Where's my package?", "expected_intent": "check_status"}
{"utterance": "I want to speak to someone", "expected_intent": "escalate"}

Minimum: 100 utterances across all intents, including edge cases and ambiguous phrases.

Step 2: Test Multi-Turn Dialogs

{"scenario": "Book meeting then change time", "turns": [
  {"user": "Book a meeting tomorrow", "expected_state": "task_execute"},
  {"user": "Actually make it 4pm instead", "expected_state": "task_execute", "context_check": "remembers 'tomorrow'"}
]}

Step 3: Measure Latency Budget

Step 4: Evaluate Voice Quality

• Run MOS test: 5 listeners rate 20 sample responses (1-5 scale)
• Check prosody: does voice match conversation context (empathetic for complaints, upbeat for confirmations)?
• Test: interrupted speech recovery (user talks over agent)
• Test: silence handling (agent waits appropriate time)

Step 5: Compare vs Play 04 (Call Center)

Step 6: Generate Report

python evaluation/eval.py --all --output evaluation/voice-report.json --ci-gate

Quality Gate Decision

Evaluation Cadence

• Pre-deployment: Full dialog scenario suite (100+ utterances)
• Weekly: Intent accuracy spot-check from production logs
• Monthly: MOS evaluation with listener panel
• On voice change: Full prosody and naturalness re-evaluation

Common Failure Patterns

CI/CD Integration

- name: Intent Accuracy Gate
  run: python evaluation/eval.py --metrics intent --ci-gate --threshold 0.92
- name: Dialog Completion Gate
  run: python evaluation/eval.py --metrics dialog --ci-gate --threshold 0.80

Source: frootai/frootai — distributed by TomeVault.