Aibench Scenario Eval

aibench-scenario-eval

AIBench Scenario: Scenario-distilling AI Benchmarking — Gao et al. (2020) (arXiv:2005.03459, 2020)

What this evaluates

Evaluates the end-to-end system-level performance and tail latency of AI-driven online services by simulating real-world user workloads. It probes how cascading interactions between AI and non-AI components affect overall service quality, and tests the validity of statistical queueing models for predicting system latency.

Datasets

• AIBench Scenario (E-commerce & Translation Intelligence) — total 20000; splits: workload (20000); repo https://github.com/BenchCouncil/aibench_scenario

Metrics

• latency (avg, p90, p99) (primary) — range: milliseconds
  • Measures the time in milliseconds to process a single query request. Reported as average, 90th percentile, and 99th percentile values across 20,000 completed requests. Also reports Coefficient of Variation (CV = standard deviation / mean) to assess reproducibility across multiple runs.

Input / output format

Input: Simulated user query requests (text, image, audio) generated by a Jmeter-based Query Generator. Think time intervals follow a Poisson distribution. Workload composition: E-commerce (99% text, 1% image), Translation (90% text, 5% image, 5% audio).

Output: System, module, and component-level latency measurements (average, 90th, 99th percentile in ms), CPU utilization (%), instructions per cycle (IPC), and queueing model prediction errors.

Scoring recipe

latencies = [get_request_latency(req) for req in requests]
avg_latency = sum(latencies) / len(latencies)
p90_latency = sorted(latencies)[int(0.9 * len(latencies))]
p99_latency = sorted(latencies)[int(0.99 * len(latencies))]
means = [get_avg_latency(run) for run in range(5)]
std_dev = sqrt(sum((m - mean(means))**2 for m in means) / 5)
cv = std_dev / mean(means)
return {"avg": avg_latency, "p90": p90_latency, "p99": p99_latency, "cv": cv}

Common pitfalls

• Assuming isolated AI component performance predicts overall system tail latency; the paper shows system tail latency can degrade hundreds of times due to cascading interactions and non-critical path components.
• Relying on simple M/M/1 queueing models to predict tail latency; the paper demonstrates these models fail because service rates do not follow exponential distributions and complex component interactions dominate.
• Ignoring communication/network latency when attributing performance bottlenecks to AI vs non-AI components.

Evidence (verbatim from paper)

The average, 90th percentile, and 99th percentile latency of the overall system of E-commerce Intelligence is 178, 238, and 316 milliseconds, respectively. Simultaneously, for Translation Intelligence, the number is 778.7, 934.4, and 5919.7 milliseconds, respectively. The two scenario benchmarks reflect different latency characteristics because of various permutations of AI and non-AI tasks.

Citation

@misc{gao2020aibenchscenario,
  title={AIBench Scenario: Scenario-distilling AI Benchmarking},
  author={Gao et al. (2020)},
  year={2020},
  note={arXiv:2005.03459}
}

• arXiv: 2005.03459