Salus Gpu Sharing Eval

salus-gpu-sharing-eval

Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications — Yu et al. (2019) (arXiv:1902.04610, 2019)

What this evaluates

Evaluates the effectiveness and overhead of fine-grained GPU sharing primitives for scheduling deep learning training, hyper-parameter tuning, and inference workloads on a single GPU.

Datasets

• Salus DL Workload Trace & Benchmarks — total ?; splits: workloads (-1); repo https://github.com/SymbioticLab/Salus

Metrics

• Makespan (primary) — range: minutes
  • Total time from the start of the first job to the completion of the last job in a set of concurrent or queued jobs.
• Average Job Completion Time (Avg. JCT) — range: minutes
  • Mean of (completion time - arrival time) across all jobs in the trace.
• GPU Utilization Improvement — range: factor
  • Ratio of GPUs required by a baseline (e.g., FIFO or MPS) to the number of GPUs required by Salus to host the same set of models concurrently.

Input / output format

Input: Deep learning models (e.g., ResNet, AlexNet, Inception) configured with specific batch sizes and durations, submitted as a chronological job trace or as a batch of concurrent jobs.

Output: Execution logs containing job start/end times, GPU memory allocation snapshots, per-iteration training times, and inference request latencies.

Scoring recipe

def evaluate_scheduler(jobs, scheduler):
    schedule = scheduler.run(jobs)
    makespan = max(j.end_time for j in schedule) - min(j.start_time for j in schedule)
    avg_jct = sum(j.end_time - j.arrival_time for j in schedule) / len(schedule)
    p95_jct = sorted(j.end_time - j.arrival_time for j in schedule)[int(0.95 * len(schedule))]
    return {'makespan': makespan, 'avg_jct': avg_jct, 'p95_jct': p95_jct}

Common pitfalls

• Overhead measurements can be skewed by CPU-bound models because Salus redirects CPU computation to its own unoptimized execution engine.
• GPU utilization gains are highly dependent on workload memory patterns (batch vs. iteration vs. idle phases) and may not generalize to compute-bound models.
• MPS comparison is limited to specific sharing scenarios and does not account for Salus's advanced scheduling policies like SRTF or FAIR.

Evidence (verbatim from paper)

We have integrated Salus with TensorFlow and evaluated it using a collection of training, hyper-parameter tuning, and inference workloads to understand its effectiveness and overhead. Table 2 presents makespan and aggregate statistics. Furthermore, packing jobs can indeed improve makespan. These experiments also reestablishes the fact that in the presence of known completion times, SRTF can indeed improve the average JCT – 3.19x w.r.t. FIFO in this case.

Citation

@misc{yu2019salus,
  title={Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications},
  author={Yu et al. (2019)},
  year={2019},
  note={arXiv:1902.04610}
}

• arXiv: 1902.04610