Focal Eval

focal-eval

FOCAL: A Novel Benchmarking Technique for Multi-modal Agents — Choudhary et al. (2026) (arXiv:2601.07367, 2026)

What this evaluates

Evaluates end-to-end reasoning, error propagation, and user experience in multi-modal cascading agents. It assesses technical performance (ASR/TTS fidelity, tool calling) and behavioral quality (reasoning, semantic similarity, contextual consistency) across simulated customer service journeys.

Datasets

• FOCAL Customer Journeys — total ?; splits: test (-1)

Metrics

• Reasoning — range: [0, 10]
  • LLM-judged score evaluating the logical soundness and step-by-step validity of the agent's response relative to the query.
• Semantic — range: [0, 10]
  • LLM-judged or embedding-based score measuring how closely the agent's response aligns semantically with the user's intent.
• Tool-Calling — range: [0, 1]
  • Binary metric indicating whether the agent correctly invoked the required tool/function for the given query.
• Accuracy Similarity (primary) — range: [0, 1]
  • Cosine similarity between embedding representations of the ground-truth/reference transcript and the agent's generated transcript.
• WER — range: [0, 1]
  • Word Error Rate measuring the ratio of insertions, deletions, and substitutions to the reference word count.
• Voice Similarity — range: [0, 1]
  • Cosine similarity between audio embedding vectors of the reference and generated voice outputs.
• MOS — range: [1, 5]
  • Mean Opinion Score assessing perceived audio quality, naturalness, and clarity on a subjective scale.
• Consistency — range: [0, 1]
  • Embedding-based similarity score measuring contextual and persona consistency across consecutive conversation turns.

Input / output format

Input: Seed queries representing customer journeys, enriched with persona data fetched from a knowledge database. The agent receives text/audio prompts simulating user requests.

Output: Multi-turn conversational responses (voice-to-voice), including tool calls, retrieved information, and spoken replies.

Scoring recipe

def compute_metrics(transcript, query, audio_gen, audio_ref, tool_calls, ground_truth_tools):
    reasoning = llm_judge_score(transcript, query, criteria='reasoning')
    semantic = llm_judge_score(transcript, query, criteria='semantic')
    tool_acc = 1 if tool_calls == ground_truth_tools else 0
    acc_sim = cosine_similarity(embed(transcript), embed(query))
    wer = compute_wer(audio_ref, audio_gen)
    voice_sim = cosine_similarity(embed(audio_ref), embed(audio_gen))
    mos = human_or_llm_score(audio_gen, criteria='quality')
    consistency = cosine_similarity(embed(turn_i), embed(turn_i_minus_1))
    return {'Reasoning': reasoning, 'Semantic': semantic, 'Tool-Calling': tool_acc,
            'Accuracy Similarity': acc_sim, 'WER': wer, 'Voice Similarity': voice_sim,
            'MOS': mos, 'Consistency': consistency}

Common pitfalls

• LLM judges for Reasoning and Semantic scores are highly sensitive to prompt templates, temperature settings, and the specific judge model used.
• WER and Voice Similarity metrics depend heavily on the ASR/TTS pipeline and audio preprocessing, making cross-system comparisons sensitive to codec and sampling rate differences.
• The Human-Simulator's random personality injection alters query difficulty and context, which can cause metric variance across otherwise identical customer journeys.

Evidence (verbatim from paper)

Different evaluation metrics viz. Mean Opinion Score (MOS), Voice quality and Accuracy for 6 different customer journeys are documented in Table [I] for the voice-to-voice architecture experimented in this work.

Citation

@misc{choudhary2026focal,
  title={FOCAL: A Novel Benchmarking Technique for Multi-modal Agents},
  author={Choudhary et al. (2026)},
  year={2026},
  note={arXiv:2601.07367}
}

• arXiv: 2601.07367