Kimi K1.5 Benchmark Eval

kimi-k1.5-benchmark-eval

Kimi k1.5: Scaling Reinforcement Learning with LLMs — Kimi Team (2025) (arXiv:2501.12599, 2025)

What this evaluates

Evaluates multimodal reasoning, coding, and instruction-following capabilities across text, code, and vision tasks using a standardized suite of academic benchmarks.

Datasets

• MMLU — total ?; splits: test (-1)
• IF-Eval — total ?; splits: test (-1)
• CLUEWSC — total ?; splits: test (-1)
• C-EVAL — total ?; splits: test (-1)
• HumanEval-Mul — total ?; splits: test (-1)
• LiveCodeBench — total ?; splits: test (-1)
• Codeforces — total ?; splits: test (-1)
• AIME 2024 — total ?; splits: test (-1)
• MATH-500 — total ?; splits: test (-1)
• MMMU — total ?; splits: val (-1)
• MATH-Vision — total ?; splits: full (-1)
• MathVista — total ?; splits: test (-1)

Metrics

• exact-match accuracy (EM) (primary) — range: [0, 1]
  • 1 if the model's generated answer exactly matches the ground truth answer, 0 otherwise. Averaged over all instances.
• pass@1 — range: [0, 1]
  • 1 if the single generated solution passes the provided test cases or matches the expected output, 0 otherwise. Averaged over all instances.
• prompt strict accuracy — range: [0, 1]
  • Exact match score under strict prompt-following constraints as defined by the IF-Eval benchmark.
• percentile — range: [0, 100]
  • Ranking percentile of the model's score relative to a reference distribution of model scores on the benchmark.

Input / output format

Input: Standard benchmark prompts for text, code, and vision-language tasks. Vision tasks include image inputs paired with text questions.

Output: Model-generated text or code responses. For code benchmarks, executable code snippets are generated.

Scoring recipe

def score_em(predictions, gold):
    return sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip()) / len(gold)

def score_pass1(predictions, gold_tests):
    passed = 0
    for pred, tests in zip(predictions, gold_tests):
        if run_and_pass(pred, tests):
            passed += 1
    return passed / len(predictions)

Common pitfalls

• Detailed evaluation setup (temperature, max tokens, sampling strategy) is deferred to Appendix C and not specified in the main text.
• Vision-language model results for baselines are sourced from the OpenCompass platform rather than directly evaluated by the authors.
• Metrics vary significantly across benchmarks (EM, Pass@1, Percentile, Prompt Strict), requiring careful alignment when comparing scores.

Evidence (verbatim from paper)

Our benchmarks primarily consist of the following three categories: Text Benchmark: MMLU*, IF-Eval*, CLUEWSC*, C-EVAL* ... Reasoning Benchmark: HumanEval-Mul, LiveCodeBench*, Codeforces, AIME 2024, MATH-500* ... Vision Benchmark: MMMU*, MATH-Vision*, MathVista* ... Table 2: Performance of Kimi k1.5 long-CoT and flagship open-source and proprietary models. (Metric) ... MATH-500 (EM) ... AIME 2024 (Pass@1) ... LiveCodeBench (Pass@1)

Citation

@misc{kimi2025k1.5,
  title={Kimi k1.5: Scaling Reinforcement Learning with LLMs},
  author={Kimi Team (2025)},
  year={2025},
  note={arXiv:2501.12599}
}

• arXiv: 2501.12599