Ml Drift Inference Eval

ml-drift-inference-eval

Scaling On-Device GPU Inference for Large Generative Models — Tang et al. (2025) (arXiv:2505.00232, 2025)

What this evaluates

Measures inference throughput and latency of large generative models across diverse on-device GPU backends. Probes the efficiency of tensor virtualization and runtime shader generation in decoupling logical semantics from physical memory layouts compared to established inference engines.

Datasets

• Stable Diffusion 1.4 — total ?; splits: test (-1)
• Gemma 2B — total ?; splits: test (-1)
• Gemma2 2B — total ?; splits: test (-1)
• Llama 3.2 3B — total ?; splits: test (-1)
• Llama 3.1 8B — total ?; splits: test (-1)

Metrics

• tokens/s (prefill) — range: other
  • Number of prefill tokens (1024) divided by the wall-clock time to process them.
• tokens/s (decode) (primary) — range: other
  • Number of generated tokens (256) divided by the wall-clock time to generate them.
• end-to-end latency (s) — range: other
  • Total wall-clock time to complete 20 sampling iterations for a 512×512 image.

Input / output format

Input: LLMs: 1280-token context (1024 prefill, 256 generation). Diffusion: 512×512 image generation with 20 sampling iterations.

Output: LLMs: sequentially generated tokens. Diffusion: final 512×512 image.

Scoring recipe

def calc_tokens_per_second(tokens, time_seconds):
    return tokens / time_seconds

def calc_end_to_end_latency(iterations, measure_iteration):
    total_time = sum(measure_iteration() for _ in range(iterations))
    return total_time

Common pitfalls

• OpenCL backends cannot access NVIDIA Tensor Cores, making prefill performance comparisons on NVIDIA GPUs invalid.
• Quantization schemes differ across baselines (ML Drift uses per-channel int8/int4, while others often use q4f16 group quantization), affecting memory bandwidth vs compute-bound characteristics.
• FP32 precision is forced on NVIDIA platforms due to OpenCL driver limitations, skewing performance relative to FP16 CUDA baselines.

Evidence (verbatim from paper)

Table 2: LLM performance (tokens/s) on Qualcomm and Arm GPUs for Gemma and Llama models. ... ML Drift’s OpenCL backend achieved a 5× to 11× speedup in token prefill compared to the other open-source LLM inference solutions.

Citation

@misc{tang2025scaling,
  title={Scaling On-Device GPU Inference for Large Generative Models},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2505.00232}
}

• arXiv: 2505.00232