Inference Latency

inference-latency

The Architectural Implications of Facebook's DNN-based Personalized Recommendation — Gupta et al. (2019) (arXiv:1906.03109, 2019)

What this evaluates

Probes how different CPU microarchitectures (Haswell, Broadwell, Skylake) and cache hierarchies affect the inference latency and throughput of production-scale DNN recommendation models under varying batch sizes and co-location scenarios.

Datasets

• Synthetic Recommendation Models (RMC1-RMC3) — total ?; splits: test (-1)

Metrics

• inference latency (primary) — range: ms
  • Measured as the average time to process a single request or batch under strict SLA constraints, typically reported in milliseconds (ms).

Input / output format

Input: Synthetic recommendation model inputs (feature vectors/embeddings) and model parameters in fp32 format, processed in non-unit batch sizes.

Output: Inference predictions (fp32), with evaluation focusing on system-level latency and throughput rather than prediction accuracy.

Scoring recipe

start = time.perf_counter()
for batch in batches:
    model.forward(batch)
end = time.perf_counter()
latency_ms = (end - start) / len(batches)
throughput_rps = len(batches) / (end - start)

Common pitfalls

• Turbo boost is explicitly disabled for all experiments, which may not reflect real-world data center behavior.
• Experiments use synthetic models and single-worker/single-thread setups, which may not capture multi-worker or production-scale concurrency effects.
• fp32 precision is used throughout, ignoring mixed-precision optimizations common in modern recsys inference.

Evidence (verbatim from paper)

Sections V and VI describe the tradeoff between the system and micro-architecture designs, and their impact on inference latency and throughput in the data center. We analyze inference performance using a benchmark which accurately represents the execution flow of production-scale models. The benchmark is implemented in Caffe2 with Intel MKL as a backend library. All experiments are run with a single Caffe2 worker and Intel MKL thread. All data and model parameters are stored in fp32 format.

Citation

@misc{gupta2019architectural,
  title={The Architectural Implications of Facebook's DNN-based Personalized Recommendation},
  author={Gupta et al. (2019)},
  year={2019},
  note={arXiv:1906.03109}
}

• arXiv: 1906.03109