Multi Agent Latency Eval

multi-agent-latency-eval

Latency and Cost of Multi-Agent Intelligent Tutoring at Scale — Elhaimeur et al. (2026) (arXiv:2604.24110, 2026)

What this evaluates

Evaluates the end-to-end latency, success rate, and cost-efficiency of a multi-agent LLM tutoring system under varying concurrency levels across different cloud inference throughput tiers. It isolates the impact of shared vs. priority vs. provisioned inference pools on response time variance and system reliability.

Datasets

• ITAS Student Query Corpus — total 100; splits: test (100)

Metrics

• end-to-end latency (primary) — range: seconds
  • Time elapsed from request submission to final response delivery. The paper emphasizes the maximum-of-parallel latency distribution and its variance under sustained concurrency.
• success_rate — range: percent
  • Percentage of requests that complete without an HTTP 500 Internal Server Error from the Vertex AI inference endpoint.

Input / output format

Input: Real student interactions from a graduate STEM seminar, including chat messages and associated code context, replayed under controlled concurrency levels.

Output: End-to-end latency, per-agent latency, parallel phase duration, token counts, bottleneck agent identification, success/failure status, and traffic type confirmation.

Scoring recipe

def compute_metrics(predictions, gold):
    latencies = [p['end_to_end_latency'] for p in predictions 
                 if p['concurrency'] == gold['target_concurrency'] and p['tier'] == gold['target_tier']]
    if not latencies: return {}
    mean_lat = sum(latencies) / len(latencies)
    var_lat = sum((x - mean_lat)**2 for x in latencies) / len(latencies)
    success_rate = sum(1 for p in predictions if p['status'] == 'success') / len(predictions)
    return {'mean_latency': mean_lat, 'latency_variance': var_lat, 'success_rate': success_rate}

Common pitfalls

• Confusing regional endpoint differences (us-east1 vs global vs us-central1) with tier performance, as baseline latencies vary by geography but scaling behavior is tier-dependent.
• Assuming standard batch-wave concurrency models apply; the benchmark uses a semaphore-based constant-concurrency model to simulate sustained classroom load.
• Overlooking that all agents ran in 'thinking-disabled mode' (thinking_budget=0), which substantially reduces latency compared to default LLM configurations.

Evidence (verbatim from paper)

Each request captures end-to-end latency, per-agent latency (video, code, guidance, synthesizer), parallel phase duration, per-agent input and output token counts, bottleneck agent identification, success/failure status, and traffic type confirmation (verifying which throughput tier actually served the request).

Citation

@misc{elhaimeur2026latency,
  title={Latency and Cost of Multi-Agent Intelligent Tutoring at Scale},
  author={Elhaimeur et al. (2026)},
  year={2026},
  note={arXiv:2604.24110}
}

• arXiv: 2604.24110