Elastic Scaling Eval

elastic-scaling-eval

Effective Elastic Scaling of Deep Learning Workloads — Saxena et al. (2020) (arXiv:2006.13878, 2020)

What this evaluates

Evaluates a deep learning job scheduler's ability to dynamically adjust GPU allocations and batch sizes to maximize cluster throughput and minimize job completion times. It probes how well the system handles compute-bound, communication-bound, and non-elastic workloads under varying job arrival patterns.

Datasets

• CIFAR100 — total ?; splits: train (-1)
• Food101 — total ?; splits: train (-1)

Metrics

• SJS Efficiency (primary) — range: other
  • Opt_Sch_Time / Act_Sch_Time, where Opt_Sch_Time is the sum of job lengths on a single GPU, and Act_Sch_Time is the sum of (actual GPUs used × time duration). Measures average scaling efficiency of scheduled jobs.
• Job Drop Ratio — range: other
  • Number of jobs dropped / Total number of jobs. Measures the proportion of jobs rejected due to resource constraints.

Input / output format

Input: Job arrival requests characterized by category (compute/communication bound, elasticity range), batch size constraints (min/max), and arrival pattern (Poisson-distributed low, high, or bursty rates).

Output: Scheduling decisions specifying the number of GPUs allocated per job and the batch size per GPU at each optimization interval.

Scoring recipe

# Calculate Opt_Sch_Time and Act_Sch_Time from scheduler logs
opt_time = sum(job_length_on_1_gpu for job in scheduled_jobs)
act_time = sum(num_gpus_allocated[job] * job_duration[job] for job in scheduled_jobs)
sjs_efficiency = opt_time / act_time

# Calculate Job Drop Ratio
drop_ratio = num_dropped_jobs / total_arrived_jobs

Common pitfalls

• Confusing GPU-only elasticity (baseline) with batch-size elasticity (proposed method).
• Applying standard system benchmarks (e.g., SPEC, Kounev metrics) instead of the paper's Opt/Act GPU time metrics.
• Assuming model accuracy or convergence speed is the primary evaluation criterion; the paper explicitly focuses on cluster throughput and job completion time.

Evidence (verbatim from paper)

We measure and report the performance in terms of two metrics (i) Scheduled Job Scaling (SJS) Efficiency, defined as Opt_Sch_Time / Act_Sch_Time, and (ii) Job Drop Ratio, defined as Number of jobs dropped / Total number of jobs.

Citation

@misc{saxena2020elastic,
  title={Effective Elastic Scaling of Deep Learning Workloads},
  author={Saxena et al. (2020)},
  year={2020},
  note={arXiv:2006.13878}
}

• arXiv: 2006.13878