Cipher Crypto Vuln Eval

cipher-crypto-vuln-eval

CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses — Manolov et al. (2026) (arXiv:2602.01438, 2026)

What this evaluates

This benchmark evaluates whether large language models can generate secure cryptographic Python code and avoid common implementation flaws under varying security guidance. It probes the model's ability to follow secure prompting instructions, correctly implement cryptographic primitives, and avoid known anti-patterns like weak hashing or fixed IVs.

Datasets

• CIPHER — total ?; splits: test (-1)

Metrics

• vulnerability_rates (primary) — range: [0, 1]
  • Proportion of generated code instances flagged with at least one vulnerability by the LLM-as-a-judge scoring pipeline. The judge identifies fine-grained vulnerability_type labels and line-level evidence spans, comparing outputs against a predefined cryptographic misuse taxonomy.

Input / output format

Input: A natural-language prompt instance from a prompt family triplet (insecure, neutral, or secure variant), optionally parameterized with algorithm choices, key sizes, or storage assumptions.

Output: A Python code snippet implementing the requested cryptographic functionality, adhering to specified interface constraints and error handling.

Scoring recipe

def compute_vulnerability_rates(predictions, judge_model, taxonomy):
    flagged_count = 0
    for code in predictions:
        # LLM-as-a-judge identifies vulnerability_type and evidence spans
        judge_output = judge_model.score(code, taxonomy)
        # Count if any target or collateral vulnerability is detected
        if judge_output.vulnerabilities:
            flagged_count += 1
    return flagged_count / len(predictions)

Common pitfalls

• Judge hallucinations or misattributed evidence spans can inflate vulnerability counts if not periodically audited by experts.
• Incomplete code that omits security-critical steps is automatically treated as vulnerable unless the prompt explicitly delegates that responsibility.
• Confusing taxonomy boundaries (e.g., 'missing authentication' vs 'weak hash') may lead to systematic judge errors if instruction prompts are not calibrated.

Evidence (verbatim from paper)

This triplet design controls for task difficulty while enabling causal comparisons of how prompting changes vulnerability rates.

Citation

@misc{manolov2026cipher,
  title={CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses},
  author={Manolov et al. (2026)},
  year={2026},
  note={arXiv:2602.01438}
}

• arXiv: 2602.01438