Flexbench Eval

flexbench-eval

Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset — Fursin et al. (2025) (arXiv:2509.11413, 2025)

What this evaluates

This evaluation probes the inference throughput and generation quality of LLMs across diverse hardware and software configurations. It also tests a predictive modeling framework designed to optimize system co-design by forecasting performance metrics based on model and hardware features.

Datasets

• OpenOrca — total ?; splits: (unstated)
• Open MLPerf Dataset — total ?; splits: (unstated); repo https://github.com/mlcommons/ck

Metrics

• ROUGE1/2/L accuracy — range: [0, 1]
  • Standard ROUGE metrics computing recall/precision of overlapping n-grams (1-gram, 2-gram) and longest common subsequence (L) between generated text and reference answers.
• Tokens/s (primary) — range: other
  • Inference throughput measured as the total number of generated tokens divided by the total inference time, reported per accelerator and overall.

Input / output format

Input: LLM prompts from the OpenOrca dataset, along with system configuration parameters (model architecture, parameter count, weight data type, framework, OS, accelerator type/count, CPU specs).

Output: Generated text completions for ROUGE scoring, and system performance logs reporting throughput (Tokens/s) and latency metrics.

Scoring recipe

def score(predictions, references, throughput_tokens, time_seconds):
    rouge_scores = {}
    for metric in ['ROUGE1', 'ROUGE2', 'ROUGEL']:
        rouge_scores[metric] = compute_rouge(predictions, references, metric)
    throughput = throughput_tokens / time_seconds
    return {'accuracy': rouge_scores, 'throughput': throughput}

Common pitfalls

• Heterogeneous hardware and software stacks require careful normalization to ensure fair cross-system comparisons.
• ROUGE metrics may poorly correlate with human-judged quality or factual correctness for complex reasoning tasks.
• Predictive models depend on cleaned, standardized features that may not capture all real-world system bottlenecks or dynamic workloads.

Evidence (verbatim from paper)

We validated our approach in the MLPerf Inference 5.0 submission by benchmarking several non-MLPerf LLMs (e.g., DeepSeek R1 Distill LLaMA 8B, LLaMA 3.3) on the OpenOrca dataset using commodity servers with NVIDIA H100 GPUs. ... A sample entry is shown below: {"metrics.accuracy": "ROUGE1: 30.6202 ROUGE2: 13.9221 ROUGEL: 18.9101 TOKENS_PER_SAMPLE: 581.8","metrics.result": 2631.93, "metrics.result_per_accelerator": 2631.93,"metrics.units": "Tokens/s", ...}

Citation

@misc{fursin2025flexbench,
  title={Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset},
  author={Fursin et al. (2025)},
  year={2025},
  note={arXiv:2509.11413}
}

• arXiv: 2509.11413