Secretbench Eval

secretbench-eval

A Comparative Study of Software Secrets Reporting by Secret Detection Tools — Basak et al. (2023) (arXiv:2307.00714, 2023)

What this evaluates

Evaluates the capability of automated secret detection tools to accurately identify hardcoded secrets (e.g., API keys, passwords, private keys) in source code repositories. It probes the tools' ability to balance high recall for true secrets against low false positive rates to mitigate alert fatigue.

Datasets

• SecretBench — total 97479; splits: test (97479); repo https://github.com/setu1421/SecretBench

Metrics

• Precision (primary) — range: [0, 1]
  • Proportion of reported secrets that are actually true secrets. Formula: TP / (TP + FP), where TP is true positives and FP is false positives.
• Recall — range: [0, 1]
  • Proportion of actual true secrets correctly identified by the tool. Formula: TP / (TP + FN), where FN is false negatives.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall. Formula: 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Source code repositories or individual files containing potential hardcoded secrets, accompanied by metadata (repository name, file path, commit ID, start line).

Output: A list of detected secrets, each reported with its location metadata (repo, path, commit, line) and the matched secret value.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    tp = fp = fn = 0
    for pred in predictions:
        match = next((gt for gt in ground_truth if matches_metadata(pred, gt)), None)
        if match and match['label'] == 'true':
            tp += 1
        else:
            fp += 1
    for gt in ground_truth:
        if gt['label'] == 'true' and not any(matches_metadata(p, gt) for p in predictions):
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• False positives often stem from generic regular expressions or flawed entropy checks that flag random strings or non-exploitable IDs as secrets.
• False negatives frequently arise from inadequate rule coverage, strict file type exclusions, or poor pattern specificity in the detection tools.
• Matching tool outputs to ground truth requires exact metadata alignment (repo, path, commit, line) rather than simple string matching, as the same secret may appear multiple times.

Evidence (verbatim from paper)

A comparative empirical evaluation of five open-source and nine proprietary secret detection tools reveals significant performance disparities in precision and recall, with Gitleaks leading in recall (88%) and GitHub Secret Scanner in precision (75%). The dataset consists of 97,479 labeled plain-text secrets (labeled as true and false) extracted from 818 repositories. We used the metadata to compare the tool-reported secrets, as discussed in Section IV.

Citation

@misc{basak2023secretbench,
  title={A Comparative Study of Software Secrets Reporting by Secret Detection Tools},
  author={Basak et al. (2023)},
  year={2023},
  note={arXiv:2307.00714}
}

• arXiv: 2307.00714