Human Communication Simulation Eval

human-communication-simulation-eval

SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems — Dong Zhang et al. (2024) (arXiv:2401.03945, 2024)

What this evaluates

Evaluates a multi-agent system's ability to generate coherent, context-aware, and emotionally expressive dialogue scripts in simulated human communication scenarios with varying numbers of roles.

Datasets

• Human-Communication Simulation Benchmark — total ?; splits: test (-1); repo https://github.com/0mutation/SpeechAgents

Metrics

• Consistency Score (primary) — range: [1, 5]
  • LLM-as-a-judge scoring on a 1–5 scale. The generated speech is transcribed via an ASR model, then evaluated by ChatGPT (GPT-4) against scene, character, and contextual descriptions using a fixed prompt template.
• Quality Score — range: [1, 5]
  • LLM-as-a-judge scoring on a 1–5 scale. Assesses language quality, emotional expression, logical consistency, and grammatical correctness of the transcribed dialogue using a separate prompt template.
• ChatGPT Score — range: [1, 5]
  • Evaluates general speech-to-speech instruction-following ability following the protocol in Zhang et al. (2023).

Input / output format

Input: Generated multi-agent dialogue scripts (speech), which are first transcribed to text via an off-the-shelf ASR model, then paired with scene/character descriptions and evaluation prompts.

Output: Numerical score (1–5) assigned by ChatGPT (GPT-4) for each evaluation dimension.

Scoring recipe

def evaluate_script(transcribed_text, context_info, prompt_template):
    full_prompt = f"{prompt_template}\nContext: {context_info}\nScript: {transcribed_text}"
    response = call_llm_api('gpt-4', full_prompt)
    score = extract_numeric_score(response, min=1, max=5)
    return score

Common pitfalls

• Relies on an external ASR model to convert generated speech to text before scoring, meaning ASR errors directly penalize the model's output.
• Scoring is entirely dependent on LLM-as-a-judge prompts (Appendices J/K) and the specific LLM version (GPT-4), making results sensitive to prompt wording and model updates.
• Evaluates textual scripts rather than raw audio, so prosodic, rhythmic, or acoustic qualities of the speech are not directly measured.

Evidence (verbatim from paper)

For human communication simulation evaluation, we use test set in Human-Communication Simulation Benchmark and utilize ChatGPT (GPT-4) as an evaluator, primarily evaluating the generated scripts from two perspectives: consistency with the scenario and characters, and the quality and logical coherence of the script content. Consistency Score evaluates whether the scripts align with the provided scene and character descriptions and contextual elements such as time and atmosphere. We leverage the off-the-shell ASR model in section 5.2 to transform the speech scripts into its corresponding text, which is subsequently submitted for evaluation. We feed the prompt in Appendix J to ChatGPT to score the model's outputs based on response quality, with scores ranging from 1 to 5.

Citation

@misc{zhang2024speechagents,
  title={SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems},
  author={Dong Zhang et al. (2024)},
  year={2024},
  note={arXiv:2401.03945}
}

• arXiv: 2401.03945