Fast Gshare Eval

fast-gshare-eval

FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference — Gu et al. (2023) (arXiv:2309.00558, 2023)

What this evaluates

Evaluates the performance of a spatio-temporal GPU sharing architecture for serverless deep learning inference. It measures how well the system manages resource multiplexing, isolation, and auto-scaling under varying workloads and allocation configurations.

Datasets

• MLPerf — total ?; splits: (unstated)

Metrics

• throughput (primary) — range: other
  • Requests per second (req/s) processed by the model under a given load.
• tail latency — range: other
  • The maximum or high-percentile latency observed per request.
• GPU utilization — range: percent
  • Percentage of GPU compute resources actively used.
• SM occupancy — range: percent
  • Percentage of Streaming Multiprocessors (SMs) actively executing threads.
• memory footprint — range: other
  • Total GPU device memory (in MB) consumed by the model and its runtime context.

Input / output format

Input: Deep learning inference requests (images, text, audio) generated by Grafana k6 load simulator, targeting specific FaaS functions (ResNet, BERT, RNNT, GNMT, ResNeXt, ViT_hug) deployed on Kubernetes with NVIDIA V100 GPUs.

Output: System metrics including throughput (req/s), tail latency (ms), GPU utilization (%), SM occupancy (%), and GPU memory footprint (MB) collected via NVIDIA DCGM-Exporter and nvidia-smi.

Scoring recipe

def compute_metrics(logs):
    throughput = logs['request_count'] / logs['duration_s']
    tail_latency = max(logs['latency_ms'])
    gpu_util = mean(logs['gpu_util_pct'])
    sm_occupancy = mean(logs['sm_occupancy_pct'])
    mem_footprint = max(logs['gpu_mem_mb'])
    slo_violations = sum(1 for l in logs['latency_ms'] if l > slo_threshold) / len(logs['latency_ms'])
    return throughput, tail_latency, gpu_util, sm_occupancy, mem_footprint, slo_violations

Common pitfalls

• Throughput scales linearly with time quota but saturates with SM partition allocation, so comparing across different spatial allocations requires careful normalization.
• Model sharing introduces a fixed 300MB overhead per model, which can increase memory usage for single-pod deployments compared to no-sharing.
• SLO violations are measured against a specific threshold (e.g., 69ms for ResNet), and auto-scaling must be triggered based on current RPS to maintain <1% violation rate.

Evidence (verbatim from paper)

The experiment is designed to evaluate the performance of spatial sharing with 100% time allocation under different spatial sharing configuration: no spatial sharing (racing), 12% partition, and 24% partition. As shown in Figure 10, as the number of pods increases, spatial sharing of 12% and 24% partitions results in significant improvements in both throughput and SM occupancy, while also reducing tail latency accordingly.

Citation

@misc{gu2023fastgshare,
  title={FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference},
  author={Gu et al. (2023)},
  year={2023},
  note={arXiv:2309.00558}
}

• arXiv: 2309.00558