Deeprecsys Eval

deeprecsys-eval

DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference — Gupta et al. (2020) (arXiv:2001.02772, 2020)

What this evaluates

Evaluates the throughput, tail latency, and power efficiency of a dynamic scheduling system for at-scale neural recommendation inference across various industry models, hardware platforms, and tail-latency constraints.

Datasets

• Industry-representative recommendation models (DLRM-RMC1/2/3, WND, MT-WND, NCF, DIN, DIEN) — total ?; splits: test (-1)

Metrics

• QPS (primary) — range: other
  • Queries Per Second, normalized to the baseline static scheduler's QPS at the low tail latency case.
• Tail Latency (p95, p99) — range: other
  • 95th and 99th percentile of inference latency, measured in milliseconds.
• QPS/Watt — range: other
  • Throughput normalized by power consumption, measuring queries processed per watt of energy.

Input / output format

Input: Recommendation inference queries with varying sizes and arrival rates, processed by neural recommendation models.

Output: System throughput (QPS), tail latency percentiles (p95, p99), and power efficiency (QPS/Watt) under different tail latency targets and hardware configurations.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of per-query inference latencies
    # gold: dict with 'time_seconds' and 'power_watts'
    qps = len(predictions) / gold['time_seconds']
    p95 = np.percentile(predictions, 95)
    p99 = np.percentile(predictions, 99)
    qps_per_watt = qps / gold['power_watts']
    return {'QPS': qps, 'p95': p95, 'p99': p99, 'QPS/Watt': qps_per_watt}

Common pitfalls

• Tail latency targets (low, medium, high) significantly impact optimal batch size and hardware offloading strategy.
• Power efficiency (QPS/Watt) does not always improve with GPU offloading; memory-intensive models may see degradation due to GPU power overhead.
• Production traffic distributions (heavier tail) differ from synthetic lognormal distributions, requiring careful batch-size tuning.

Evidence (verbatim from paper)

Overall, DeepRecSched-CPU achieves 1.7×, 2.1×, and 2.7× higher QPS across all models for the low, medium, and high tail latency targets, respectively. DeepRecSched-CPU is able to increase the overall system throughput by operating at the optimal batch size configuration. Furthermore, DeepRecSched-GPU increases performance improvement to 4.0×, 5.1×, and 5.8× at the low, medium, and high tail latency targets, respectively.

Citation

@misc{gupta2020deeprecsys,
  title={DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference},
  author={Gupta et al. (2020)},
  year={2020},
  note={arXiv:2001.02772}
}

• arXiv: 2001.02772