P50 Latency

p50-latency

A CPU-Centric Perspective on Agentic AI — Ritik Raj et al. (2025) (arXiv:2511.00739, 2025)

What this evaluates

Evaluates the end-to-end latency and energy efficiency of CPU/GPU scheduling strategies for agentic AI workloads under batched request arrivals.

Datasets

• LangChain — total ?; splits: test (-1)
• Haystack RAG — total ?; splits: test (-1)
• SWE-Agent — total ?; splits: test (-1)

Metrics

• P50 latency (primary) — range: seconds
  • 50th percentile of request completion times across a batch.
• P90 latency — range: seconds
  • 90th percentile of request completion times.
• P99 latency — range: seconds
  • 99th percentile of request completion times.
• CPU dynamic energy — range: joules
  • Total energy consumed by CPU during execution, calculated as power multiplied by execution time.

Input / output format

Input: Agentic AI workloads (RAG queries, code execution, web search) submitted as closed-loop batches (B=128 or B=256) arriving simultaneously at t=0.

Output: Per-request completion timestamps aggregated into P50/P90/P99 latency values and total CPU dynamic energy.

Scoring recipe

def compute_metrics(completion_times, cpu_power_profile):
    sorted_times = sorted(completion_times)
    p50 = sorted_times[len(sorted_times) // 2]
    p90 = sorted_times[int(len(sorted_times) * 0.9)]
    p99 = sorted_times[int(len(sorted_times) * 0.99)]
    energy = sum(cpu_power_profile) * max(completion_times)
    return {'p50': p50, 'p90': p90, 'p99': p99, 'energy': energy}

Common pitfalls

• Evaluates only single-run times despite noting ~5% statistical variance across multiple runs.
• Assumes a closed-loop arrival model where all B requests arrive simultaneously at t=0, unlike real-world streaming traffic.
• Results are hardware-specific to Intel Emerald Rapids CPU and B200 GPU; scaling to other architectures requires re-evaluation.

Evidence (verbatim from paper)

We assume a closed-loop arrival system (all $B$ requests arrive simultaneously at t=0) to evaluate CGAM ([Section 6.2]) and MAWS ([Section 6.3]) for $B=128$ and combined MAWS+CGAM ([Section 6.4]) for $B=256. ... CGAM results in 2.11$\times$ (11.21 s to 5.32 s), 1.94$\times$ (42.87 s to 22.12 s) and 1.72$\times$ (65.08 s to 37.82 s) respective reduction in P50 latency ... MAWS performs 1.17$\times$ better in terms of P99 latency while maintaining similar P50 latency.

Citation

@misc{raj2025cpucentric,
  title={A CPU-Centric Perspective on Agentic AI},
  author={Ritik Raj et al. (2025)},
  year={2025},
  note={arXiv:2511.00739}
}

• arXiv: 2511.00739