Drv Code Security Eval

drv-code-security-eval

Detect--Repair--Verify for LLM-Generated Code: A Multi-Language, Multi-Granularity Empirical Study — Cheng et al. (2026) (arXiv:2603.23633, 2026)

What this evaluates

Evaluates the effectiveness of a Detect-Repair-Verify (DRV) workflow for fixing security vulnerabilities in LLM-generated code across different programming languages and granularity scopes (project, requirement, file). It measures how well iterative repair converges to a state that is both functionally correct and secure.

Datasets

• Custom LLM-generated code artifacts (JS, PHP, Python) — total ?; splits: test (-1); repo https://github.com/Hahappyppy2024/EmpiricalVDR

Metrics

• Used Iterations — range: integer
  • Records the iteration number at which the reported result is obtained. Lower values indicate faster convergence.
• Func. Pass Rate — range: [0, 1]
  • Proportion of repaired artifacts that preserve expected functional behavior as verified by test suites.
• Exploit Success Rate — range: [0, 1]
  • Proportion of repaired artifacts where the corresponding exploit remains successful. Lower values indicate better security improvement.
• S\C Yield Rate (primary) — range: [0, 1]
  • Records whether a case is both functionally correct and non-exploitable. Calculated as the average of (Func. Pass Rate == 1.0 AND Exploit Success Rate == 0.0) across cases. Serves as the main indicator of secure-and-correct convergence.

Input / output format

Input: LLM-generated code artifacts at project, requirement, or file granularity, accompanied by known security vulnerabilities, functional test suites, and exploit scripts.

Output: Repaired code artifacts. Each artifact is evaluated to produce binary/continuous scores for functional pass rate, exploit success rate, and secure-and-correct yield.

Scoring recipe

def score_artifact(repaired_code, tests, exploits):
    func_pass = run_tests(repaired_code) == PASS
    exploit_success = run_exploits(repaired_code) == SUCCESS
    func_pass_rate = 1.0 if func_pass else 0.0
    exploit_success_rate = 1.0 if exploit_success else 0.0
    sc_yield = 1.0 if (func_pass and not exploit_success) else 0.0
    return func_pass_rate, exploit_success_rate, sc_yield

def aggregate_metrics(results):
    avg_func = sum(r[0] for r in results) / len(results)
    avg_exploit = sum(r[1] for r in results) / len(results)
    avg_sc = sum(r[2] for r in results) / len(results)
    return avg_func, avg_exploit, avg_sc

Common pitfalls

• Repair scope (project vs. requirement vs. file) drastically changes convergence and success rates; results are not directly comparable across granularities without normalization.
• Exploit Success Rate measures residual vulnerability, not detection accuracy; a low rate does not guarantee the vulnerability was correctly identified or patched.
• S\C Yield Rate requires simultaneous optimization of functionality and security; optimizing solely for functional pass rate may leave vulnerabilities intact.

Evidence (verbatim from paper)

Four indicators are used throughout. Used Iterations records the iteration at which the reported result is obtained. Func. Pass Rate shows whether the repaired artifact still preserves the expected functional behavior. Exploit Success Rate shows whether the corresponding exploit remains successful after repair, where lower values indicate better security improvement. S\C Yield Rate records whether a case is both functionally correct and non-exploitable, and serves as the main indicator of secure-and-correct convergence.

Citation

@misc{cheng2026detectrepairverify,
  title={Detect--Repair--Verify for LLM-Generated Code: A Multi-Language, Multi-Granularity Empirical Study},
  author={Cheng et al. (2026)},
  year={2026},
  note={arXiv:2603.23633}
}

• arXiv: 2603.23633