Mobilekernelbench Eval

mobilekernelbench-eval

MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices? — Zou et al. (2026) (arXiv:2603.11935, 2026)

What this evaluates

Evaluates LLMs' capability to generate syntactically valid, functionally correct, and hardware-efficient C/C++ kernels for mobile inference engines. It probes framework-specific API usage, compilation robustness, functional verification against ONNX baselines, and runtime speedup optimization.

Datasets

• MobileKernelBench — total ?; splits: train (150), test (40); repo https://github.com/alibaba/MNN

Metrics

• Compilation success rate (CSR) (primary) — range: percent
  • Percentage of generated kernels that successfully pass the build/compilation process for the target MNN CPU backend.
• Functional correctness rate (FCR) — range: percent
  • Percentage of generated kernels that pass functional verification against the ONNX baseline implementation.
• fastp — range: percent
  • Percentage of tasks where the generated kernel achieves a speedup greater than threshold p (e.g., 0.5, 1.0, 1.5) relative to the native MNN implementation. Reported for the best-performing candidate per task.

Input / output format

Input: Natural language description of the target operator/kernel, typically accompanied by an ONNX baseline or reference implementation, and framework-specific constraints for the MNN inference engine.

Output: C/C++ source code implementing the MNN operator/kernel, including necessary registration and execution logic.

Scoring recipe

# For each task in the test set:
# 1. Attempt to compile generated code against MNN CPU backend.
# 2. If compilation succeeds, run functional verification against ONNX baseline.
# 3. If functionally correct, benchmark runtime speedup vs native MNN implementation.
# 4. For iterative/multi-candidate methods, keep only the best-performing kernel per task.

CSR = (sum(1 for task in tasks if task.compile_success) / len(tasks)) * 100
FCR = (sum(1 for task in tasks if task.functional_verify_success) / len(tasks)) * 100
fastp = (sum(1 for task in tasks if task.speedup > p) / len(tasks)) * 100

Common pitfalls

• Base LLMs frequently hallucinate framework-specific APIs or dependencies, leading to compilation failure rates exceeding 50%.
• Syntactic validity does not guarantee functional correctness; strict verification against ONNX baselines is required to catch logical errors.
• Performance optimization (fastp) is extremely difficult for standard prompting or basic fine-tuning without iterative feedback or external tooling.

Evidence (verbatim from paper)

We assess performance on MobileKernelBench using three metrics: (1) Compilation success rate (CSR), the percentage of tasks where generated operators successfully pass the build process. (2) Functional correctness rate (FCR), the proportion of tasks that pass functional verification against the ONNX baseline. (3) fastp, this metric quantifies the percentage of tasks achieving a speedup greater than a threshold p relative to the native MNN implementation. For methods that involve iterative refinement or generate multiple candidates, we report metrics based on the best-performing operator for each task.

Citation

@misc{zou2026mobilekernelbench,
  title={MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?},
  author={Zou et al. (2026)},
  year={2026},
  note={arXiv:2603.11935}
}

• arXiv: 2603.11935