Bench360 Eval

bench360-eval

Bench360: Benchmarking Local LLM Inference from 360{\deg} — Stuhlmann et al. (2025) (arXiv:2511.16682, 2025)

What this evaluates

Evaluates local LLM inference across multiple dimensions, including task-specific quality (e.g., accuracy, F1, ROUGE) and system-level performance (latency, throughput, energy, memory, cold-start) under simulated workloads (single-stream, batch, server).

Datasets

• mmlu — total ?; splits: test (-1)
• squad_v2 — total ?; splits: test (-1)
• cnn_dailymail — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted answers compared to reference answers.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for token/word matching against reference answers.
• ROUGE — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation, measuring n-gram overlap between generated and reference text.
• TTFT — range: other
  • Time elapsed from prompt submission to the first generated token.
• TPOT — range: other
  • Average time taken to generate each output token.
• TPS — range: other
  • Number of tokens generated per second.
• Energy per token — range: other
  • Total GPU energy consumed divided by the number of generated tokens.

Input / output format

Input: Standardized prompts generated from a dataset via the Task Engine, configured via YAML (task type, dataset, model, backend, quantization, workload scenario).

Output: Generated text completion from the LLM, plus runtime traces (timing, resource usage) captured by the Metrics Collector.

Scoring recipe

def score(predictions, references, traces):
    acc = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip()) / len(predictions)
    f1 = compute_f1(predictions, references)
    rouge = compute_rouge(predictions, references)
    ttft = [t['ttft'] for t in traces]
    tpot = [t['tpot'] for t in traces]
    tps = [t['tokens'] / t['gen_time'] for t in traces]
    energy_per_tok = sum(t['energy_wh'] * 3600 for t in traces) / sum(t['tokens'] for t in traces)
    return {'accuracy': acc, 'F1': f1, 'ROUGE': rouge, 'TTFT': mean(ttft), 'TPOT': mean(tpot), 'TPS': mean(tps), 'Energy_per_token': energy_per_tok}

Common pitfalls

• In server mode, the benchmark does not impose artificial queuing or batching; observed latency and throughput depend entirely on the backend engine's internal scheduling and capacity constraints.
• Energy measurement relies exclusively on GPU telemetry via NVML, which may not capture CPU, memory, or cooling system power draw, potentially underestimating total system energy consumption.
• Cold-start latency aggregates container startup, model loading, and TTFT, making it difficult to isolate whether delays stem from the inference engine, Docker runtime, or model initialization.

Evidence (verbatim from paper)

We measure latency at different granularities including time-to-first-token (TTFT), time per output token (TPOT), and generation latency (GL). Throughput is tracked in tokens per second (TPS). These metrics are recorded during generation using consistent timing hooks across all backends.

Citation

@misc{stuhlmann2025bench360,
  title={Bench360: Benchmarking Local LLM Inference from 360{\deg}},
  author={Stuhlmann et al. (2025)},
  year={2025},
  note={arXiv:2511.16682}
}

• arXiv: 2511.16682