Dualgauge Eval

dualgauge-eval

DUALGUAGE: Automated Joint Security-Functionality Benchmarking for Secure Code Generation — Pathak et al. (2025) (arXiv:2511.20709, 2025)

What this evaluates

Probes the joint functional correctness and security of LLM-generated code, while also evaluating an automated framework's ability to execute code in sandboxes and semantically judge test outcomes against human ground truth.

Datasets

• DualGauge-Bench — total 154; splits: test (154)

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed over binary pass/fail decisions or trace alignment outcomes.
• Precision — range: [0, 1]
  • Ratio of true positive decisions (correctly predicted pass/fail or trace matches) to all positive predictions.
• Recall — range: [0, 1]
  • Ratio of true positive decisions to all actual positive cases in the ground truth.

Input / output format

Input: Code generation prompts, paired functional and security test cases, and generated code samples from target LLMs.

Output: Execution traces/outputs from the agentic executor, and binary pass/fail decisions from the LLM-based evaluator.

Scoring recipe

# For each sampled scenario:
# 1. Run agentic executor on code + test input -> get predicted trace/output
# 2. Compare predicted trace/output to manually verified ground truth -> binary match (1 if aligned, 0 otherwise)
# 3. Run LLM evaluator on code + test input -> get predicted pass/fail
# 4. Compare predicted pass/fail to ground truth pass/fail -> binary match (1 if correct, 0 otherwise)
# 5. Compute precision, recall, F1 across all 154 samples
tp = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, ground_truth) if p == 0 and g == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Structural misalignment in execution traces (e.g., missing intermediate steps, extra reasoning text, or reordered actions) can cause false negatives even if the final output is correct.
• Strict or ambiguous evaluation criteria may cause the LLM-based evaluator to misinterpret acceptable behavioral variations as errors.

Evidence (verbatim from paper)

Using this random sample, we first evaluate the agentic executor. We manually review the execution trace and output for each sampled scenario, determining the correct behavior/output that should result from executing the candidate program under the chosen test input. We compare these ground-truth traces to those produced by our agentic executor and compute precision, recall, and F1.

Citation

@misc{pathak2025dualgauge,
  title={DUALGUAGE: Automated Joint Security-Functionality Benchmarking for Secure Code Generation},
  author={Pathak et al. (2025)},
  year={2025},
  note={arXiv:2511.20709}
}

• arXiv: 2511.20709