Dllm Se Eval

dllm-se-eval

Exploring the Power of Diffusion Large Language Models for Software Engineering: An Empirical Investigation — Jingyao Zhang et al. (2025) (arXiv:2510.04605, 2025)

What this evaluates

Evaluates the effectiveness and efficiency of Diffusion LLMs versus Autoregressive LLMs across the software development lifecycle. It probes code generation accuracy, binary defect detection, automated program repair, and cross-file issue resolution, while measuring generation throughput and latency.

Datasets

• HumanEval — total ?; splits: test (-1)
• Mercury — total ?; splits: Easy (-1), Medium (-1), Hard (-1)
• Devign — total ?; splits: test (-1)
• Bears — total 251; splits: Detection (-1), Repair (-1)
• Defects4J — total ?; splits: test (-1)
• SWE-bench — total ?; splits: test (-1)

Metrics

• Pass@K (primary) — range: [0, 1]
  • Samples K code generations per prompt. The metric calculates the fraction of prompts where at least one generated solution passes all provided unit tests.
• DDF1 — range: [0, 1]
  • Macro-averaged F1 score for binary defect detection classification, computed as the unweighted mean of recall and precision across classes.
• PR — range: percent
  • Program Repair success rate, defined as the percentage of generated patches that successfully compile and pass validation tests out of total attempts.
• MDVR / PRR — range: percent
  • Multi-File Defect Verification Rate and Patch Resolution Rate for SWE-bench, measuring the percentage of benchmark issues where the model's patch correctly resolves the reported bug.
• TPS / T_avg — range: other
  • Tokens Per Second (TPS) measures generation throughput; T_avg measures average inference time per task instance.

Input / output format

Input: Code generation prompts, bug descriptions, or issue reports requiring code completion, defect classification, or patch generation.

Output: Generated code snippets, binary defect labels, or code patches.

Scoring recipe

def compute_pass_at_k(generations, tests, k=1):
    passed = [any(run_test(gen, tests) for gen in gens) for gens in generations]
    return sum(passed) / len(passed)

def compute_pr(patches, validation_suite):
    success = sum(1 for p in patches if compile_and_run(p, validation_suite))
    return success / len(patches) * 100

def compute_ddf1(labels_true, labels_pred):
    return macro_f1_score(labels_true, labels_pred)

Common pitfalls

• Bears dataset has only 251 instances with 75% class imbalance, making macro-F1 (DDF1) unreliable and prone to statistical artifacts.
• Efficiency metrics (TPS, T_avg) are hardware-dependent and should not be compared across different experimental setups or hardware configurations.
• Pass@K requires sampling multiple solutions; using a single generation underestimates model capability and misrepresents the reported metric.

Evidence (verbatim from paper)

Across the full SDLC, Diff-Mercury-7B systematically outperforms the equally-sized AR-Llama3-8B Table [1]-[5]. The margin increases with task difficulty: HumanEval Pass@1 rises by 36%, Mercury-Hard by 46%, and SWE-bench PRR doubles to 32% while PCR gains 20.7 pp; Defects4J yields five additional compilable patches per 100 attempts and Bears-repair success climbs 2.6 times.

Citation

@misc{zhang2025exploring,
  title={Exploring the Power of Diffusion Large Language Models for Software Engineering: An Empirical Investigation},
  author={Jingyao Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.04605}
}

• arXiv: 2510.04605