Sv Trusteval C Eval

sv-trusteval-c-eval

SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis — Yansong Li et al. (arXiv:2505.20630, 2025)

What this evaluates

Evaluates LLMs' structural and semantic reasoning capabilities in C code vulnerability analysis. It probes whether models rely on memorized patterns or genuinely understand code interdependencies by testing consistency across base, data-flow, control-flow, counterfactual, goal-driven, and predictive scenarios.

Datasets

• SV-TrustEval-C — total ?; splits: test (-1)

Metrics

• Cons_DFL (primary) — range: [0, 1]
  • Calculates the proportion of cases where the model is correct on both the base scenario and the DataFlow-wise variant. Formula: sum(I(C_base=1 ∧ C_DFL=1)) / N_DFL.
• Cons_CFL — range: [0, 1]
  • Calculates the proportion of cases where the model is correct on both the base scenario and the ControlFlow-wise variant. Formula: sum(I(C_base=1 ∧ C_CFL=1)) / N_CFL.
• Cons_CTF — range: [0, 1]
  • Calculates the proportion of cases where the model is correct on both the base scenario and the Counterfactual variant. Formula: sum(I(C_CTF=1 ∧ C_base=1)) / N_CTF.
• Cons_GDV — range: [0, 1]
  • Calculates the proportion of cases where the model is correct on both the safe base scenario and the Goal-driven variant. Formula: sum(I(C_GDV=1 ∧ C_safe=1)) / N_GDV.
• Cons_PRD — range: [0, 1]
  • Calculates the proportion of cases where the model is correct on both the unsafe base scenario and the Predictive variant. Formula: sum(I(C_PRD=1 ∧ C_unsafe=1)) / N_PRD.

Input / output format

Input: C code snippet (with vulnerability annotations, function names, and variable names masked) paired with a multiple-choice question. In-context learning prompts additionally include 1-3 demonstration Q&A pairs with explanations.

Output: A single letter choice (A, B, C, or D) corresponding to the answer options, or a vulnerability classification label. Generation is constrained to a maximum of 50 tokens.

Scoring recipe

def compute_consistency(base_correct, variant_correct):
    # base_correct and variant_correct: list of 1/0 indicating correctness
    numerator = sum(b and v for b, v in zip(base_correct, variant_correct))
    return numerator / len(base_correct)

Common pitfalls

• Label masking replaces vulnerability-specific names and annotations, so models cannot rely on memorized Juliet test patterns.
• In-context learning improves performance for specialized code models but can degrade general-purpose models (e.g., GPT-4) in specific semantic reasoning tasks.
• High false positive rates occur when models incorrectly classify safe code as unsafe, often reaching near 100% for some architectures.

Evidence (verbatim from paper)

For each question derived from the base scenario, we define the consistency scores as: $$ \operatorname {C o n s } _ {\mathrm {D F L}} = \frac {\sum_ {i = 1} ^ {N _ {\mathrm {D F L}}} \mathbb {I} \left(C _ {\text {b a s e}} ^ {i} = 1 \wedge C _ {\mathrm {D F L}} ^ {i} = 1\right)}{N _ {\mathrm {D F L}}} $$ where: $\mathbb{I}(\cdot)$ is the indicator function, which equals 1 if the condition inside is true, and 0 otherwise. $C_{\text{safe}}^i$ and $C_{\text{unsafe}}^i$ denote the correctness indicators for safe and unsafe classifications in the base scenario for the $i$-th case.

Citation

@misc{li2025svtrustealc,
  title={SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis},
  author={Yansong Li et al.},
  year={2025},
  note={arXiv:2505.20630}
}

• arXiv: 2505.20630