Kernelbench Eval

kernelbench-eval

KernelBench: Can LLMs Write Efficient GPU Kernels? — Ouyang et al. (2025) (arXiv:2502.10517, 2025)

What this evaluates

Evaluates large language models' ability to generate functionally correct and hardware-efficient CUDA kernels for PyTorch workloads. It probes the models' capacity for low-level systems programming, hardware-aware optimization, and debugging execution/functional errors under one-shot prompting.

Datasets

• KernelBench — total ?; splits: test (-1)

Metrics

• fast_p (primary) — range: percent
  • Percentage of problems where the model-generated kernel is functionally correct and achieves at least a p-times speedup over the PyTorch Eager baseline. At p=1, it measures the percentage of kernels faster than PyTorch Eager.

Input / output format

Input: A PyTorch model/task to optimize, accompanied by one in-context example of a simple add operator kernel showing the expected input/output format.

Output: A CUDA kernel implementation (ModelNew) written as source code.

Scoring recipe

def compute_fast_p(predictions, baselines, p=1.0):
    correct_count = 0
    for pred in predictions:
        if is_functionally_correct(pred, baselines['gold']):
            pred_time = profile_on_gpu(pred, device='L40S')
            base_time = baselines['pytorch_eager_time']
            if pred_time <= base_time / p:
                correct_count += 1
    return (correct_count / len(predictions)) * 100

Common pitfalls

• Confusing fast_p with a continuous speedup ratio; it is a thresholded percentage metric that only counts kernels meeting both correctness and speed thresholds.
• Assuming hardware portability; kernels optimized for one GPU (e.g., L40S) often show significant speedup variations on different architectures (e.g., A10G).
• Overlooking that fast_0 counts all functionally correct kernels regardless of speed, while fast_1 strictly requires >1x speedup over the baseline.

Evidence (verbatim from paper)

We profile the generated code on an NVIDIA L40S GPU, and measure the $ ext{fast}{p}$ metric across all problems. Figure 4 shows the distribution of $ ext{fast}{p}$ as $p$ varies, indicating the percentage of kernels that are $p$-times faster than the PyTorch Eager baseline (the top right of the plot is better).

Citation

@misc{ouyang2025kernelbench,
  title={KernelBench: Can LLMs Write Efficient GPU Kernels?},
  author={Ouyang et al. (2025)},
  year={2025},
  note={arXiv:2502.10517}
}

• arXiv: 2502.10517