Cweval Eval

cweval-eval

CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation — Jinjun Peng et al. (2025) (arXiv:2501.08200, 2025)

What this evaluates

Evaluates whether LLM-generated code is simultaneously functionally correct and secure against vulnerabilities. It measures the pass rate for functionality alone versus the joint pass rate for functionality and security, highlighting the gap where models produce working but vulnerable code.

Datasets

• CWEVAL-BENCH — total ?; splits: test (-1); repo https://github.com/Collin/CWEval

Metrics

• func@k — range: percent
  • Percentage of tasks where at least one of the top-k sampled code generations passes the functionality test suite.
• func-sec@k (primary) — range: percent
  • Percentage of tasks where at least one of the top-k sampled code generations passes both the functionality test suite and dynamic security tests (or reference implementation checks).

Input / output format

Input: Task description or specification for a coding problem.

Output: Generated code snippet.

Scoring recipe

def compute_metrics(generated_samples, func_tests, sec_tests, k):
    func_pass_count = 0
    func_sec_pass_count = 0
    for samples, f_test, s_test in zip(generated_samples, func_tests, sec_tests):
        top_k = samples[:k]
        has_func = any(f_test.run(code) for code in top_k)
        has_sec = any(s_test.run(code) for code in top_k)
        if has_func: func_pass_count += 1
        if has_func and has_sec: func_sec_pass_count += 1
    total = len(generated_samples)
    return func_pass_count / total, func_sec_pass_count / total

Common pitfalls

• Evaluating functionality and security separately fails to capture the 'alignment tax' where models learn to avoid security-sensitive code entirely to maximize security scores, degrading helpfulness.
• Relying solely on static analysis tools can yield false positives/negatives; this benchmark requires dynamic execution oracles and reference implementations for accurate validation.
• Assuming higher functional pass rates guarantee security; the paper shows a consistent ~30% drop from func@k to func-sec@k across models.

Evidence (verbatim from paper)

We observe that for all LLMs, there is a significant performance gap between only functionality pass rate and pass rate requiring both functionality and security. From func@10 to func/sec@10, the performance drops around 30% across all models, with the maximum 35.79% observed on Gemini 1.5 Flash. This shows that in security-critical coding scenarios, LLMs often generate functional but insecure code with vulnerability issues, which is very likely to be ignored by developers and introduce serious potential risk.

Citation

@misc{peng2025cweval,
  title={CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation},
  author={Jinjun Peng et al. (2025)},
  year={2025},
  note={arXiv:2501.08200}
}

• arXiv: 2501.08200