Tally Gpu Sharing Eval

tally-gpu-sharing-eval

Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads — Zhao et al. (2024) (arXiv:2410.07381, 2024)

What this evaluates

Evaluates the performance isolation and resource sharing capabilities of GPU scheduling systems for concurrent deep learning workloads. It probes how well a system maintains tail latency for high-priority inference tasks while maximizing throughput for best-effort training tasks under varying traffic loads and workload combinations.

Datasets

• Tally Benchmark Suite — total ?; splits: test (-1); repo https://github.com/tally-project/tally-bench.git

Metrics

• 99th-percentile latency (primary) — range: other
  • The 99th percentile of inference request completion times. Throughput is defined as the number of samples processed per unit time for training, or requests per minute for inference. System throughput is calculated by summing the normalized throughput of all concurrent workloads.

Input / output format

Input: Concurrent deep learning workloads (training or inference) executed on a single GPU. Inference workloads are driven by the MAF2 trace adapted to specific traffic loads (10%–90% idle time). Training workloads run continuously.

Output: Measured 99th-percentile inference latency, per-workload throughput, and aggregated system throughput. Overhead percentages for virtualization and kernel transformation.

Scoring recipe

p99_latency = np.percentile(inference_latencies, 99)
throughput = total_samples_processed / total_execution_time
system_throughput = sum(normalized_throughput(w) for w in concurrent_workloads)
overhead_pct = ((baseline_metric - tally_metric) / baseline_metric) * 100

Common pitfalls

• Coarse-grained kernel-level scheduling (e.g., TGS, MPS) fails to mitigate interference from long-running best-effort kernels, causing severe tail latency degradation.
• MAF2 trace must be adapted to match the specific inference latency of the target model to accurately simulate traffic load.
• System throughput is calculated as the sum of normalized throughputs, not raw throughput, to account for different workload types.

Evidence (verbatim from paper)

We use the $99^{th}$ percentile latency as the primary metric for assessing the performance of inference tasks. For both training and inference tasks, we measure their throughput, defined as the number of samples processed per unit of time. To evaluate the overall performance of concurrent task execution, we calculate system throughput by summing the normalized throughput of all concurrent workloads.

Citation

@misc{zhao2024tally,
  title={Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2410.07381}
}

• arXiv: 2410.07381