Meta Metrics Inference Bench

meta-metrics-inference-bench

Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — Salaria et al. (2025) (arXiv:2508.10251, 2025)

What this evaluates

Evaluates the efficiency and representational fidelity of LLM inference benchmarking methodologies by quantifying how well a reduced set of experimental parameters can accurately predict system performance compared to exhaustive testing. It measures the trade-off between computational cost and the accuracy of projected latency and throughput metrics.

Datasets

• Synthetic Inference Workloads — total ?; splits: (unstated)

Metrics

• ttft — range: milliseconds
  • Time to First Token: measures the latency from sending a prompt to receiving the first generated token. Independent of output length.
• itl — range: milliseconds
  • Inter-Token Latency: measures the average time between subsequent generated tokens during autoregressive decoding. Varies with output length and batch size.
• efficiency_metric (primary) — range: other (ratio/factor)
  • E(G, P) = (1 - Δ(G, P)) / (C_P / C_G), where Δ is the global accuracy factor (e.g., average normalized relative difference) between ground truth measurements G and projected measurements P, and C represents total experimental cost.

Input / output format

Input: Control parameters: model architecture, precision, parallelism configuration, inference backend, input sequence length, output generation length, and batch size.

Output: Performance measurements: Time to First Token (TTFT), Inter-Token Latency (ITL), Throughput (THP in tokens/sec), and Cost per million tokens ($/M).

Scoring recipe

def compute_efficiency_metric(G_metrics, P_metrics, C_G, C_P):
    delta_values = []
    for key in G_metrics:
        f_G = G_metrics[key]
        f_P = P_metrics[key]
        delta = abs(f_G - f_P) / (f_G + f_P)
        delta_values.append(delta)
    delta_global = sum(delta_values) / len(delta_values)
    efficiency = (1 - delta_global) / (C_P / C_G)
    return efficiency

Common pitfalls

• Assuming linear interpolation accurately captures non-linear GPU saturation effects at high batch sizes without empirical validation.
• Confusing system-level inference metrics (TTFT, ITL, throughput) with model accuracy/quality metrics, which are explicitly out of scope for this framework.
• Ignoring hardware-specific memory constraints when scaling batch sizes or comparing tensor parallelism configurations across different GPU counts.

Evidence (verbatim from paper)

A simple efficiency metric E can be defined as: E(G, P) = (1 - Δ(G, P)) / (C_P / C_G)

Citation

@misc{salaria2025metametrics,
  title={Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking},
  author={Salaria et al. (2025)},
  year={2025},
  note={arXiv:2508.10251}
}

• arXiv: 2508.10251