Ctibench Eval

ctibench-eval

CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence — Md Tanvirul Alam et al. (2024) (arXiv:2406.07599, 2024)

What this evaluates

Evaluates large language models on five cyber threat intelligence (CTI) tasks, including knowledge recall, vulnerability mapping, CVSS scoring, attack technique extraction, and threat attribution. It probes factual accuracy, logical reasoning, and contextual understanding within a domain-specific cybersecurity context.

Datasets

• CTIBench — total ?; splits: test (-1); repo https://github.com/xashru/cti-bench

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correct predictions out of total instances. Used for CTI-MCQ and CTI-RCM tasks.
• MAD — range: [0, 10]
  • Mean Absolute Deviation between ground truth CVSS v3.1 scores and model-predicted scores. Scores are deterministically derived from predicted vector strings using the cvss Python library.
• Micro-F1 — range: [0, 1]
  • Micro-averaged F1 score for multi-label classification, capturing both precision and recall across all extracted MITRE ATT&CK technique IDs.
• Correct Accuracy — range: [0, 1]
  • Fraction of predictions correctly identifying the threat actor or alias.
• Plausible Accuracy — range: [0, 1]
  • Fraction of predictions classified as either correct or plausible (where the report lacks sufficient details but the model provides a related actor).

Input / output format

Input: Zero-shot prompt containing an instruction to act as a cybersecurity expert, followed by task-specific context (e.g., CVE description, vulnerability string, threat report text, or historical pattern data).

Output: Task-dependent free-text response. For mapping/scoring tasks, the model must provide a justification and ensure the final line contains only the target identifier (e.g., CWE ID, CVSS vector string, or MITRE ATT&CK ID).

Scoring recipe

def score_ctibench(predictions, golds, task):
    if task in ['CTI-MCQ', 'CTI-RCM']:
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'CTI-VSP':
        pred_scores = [cvss.parse(p).score for p in predictions]
        return sum(abs(p - g) for p, g in zip(pred_scores, golds)) / len(golds)
    elif task == 'CTI-ATE':
        tp = fp = fn = 0
        for p, g in zip(predictions, golds):
            p_set, g_set = set(p), set(g)
            tp += len(p_set & g_set)
            fp += len(p_set - g_set)
            fn += len(g_set - p_set)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif task == 'CTI-TAA':
        correct = sum(1 for p in predictions if classify(p) == 'correct')
        plausible = sum(1 for p in predictions if classify(p) == 'plausible')
        return correct / len(predictions), (correct + plausible) / len(predictions)

Common pitfalls

• For CTI-VSP, models may incorrectly compute CVSS scores from vector strings; the protocol mandates using the cvss Python library to parse predictions and eliminate LLM computation errors.
• CTI-TAA evaluation requires human judgment to categorize predictions as correct, plausible, or incorrect, making full automation difficult without the paper's specific annotation guidelines.
• Zero-shot prompts require strict formatting compliance (e.g., 'last line contains only the CWE ID'); models often fail to adhere, necessitating robust regex parsing for evaluation.

Evidence (verbatim from paper)

We use accuracy to evaluate the CTI-MCQ and CTT-RCM tasks, as both tasks are equivalent to multi-class classification. For the CTI-VSP task, we compute the mean absolute deviation (MAD) between the CVSS v3.1 scores of the ground truth and the model's predictions. Although we ask the model to predict a vector string, the CVSS score can be deterministically derived from it. We utilize the Python library cvss [42] to compute the CVSS score (a numerical value in the range of 0-10 that determines the overall severity of a vulnerability) from the predicted string, ensuring that any potential errors from the LLM performing the computation are eliminated. This approach focuses solely on assessing the LLM's reasoning ability regarding vulnerability. We adopt the Micro-F1 score as the evaluation metric for the CTI-ATE task.

Citation

@misc{alam2024ctibench,
  title={CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence},
  author={Md Tanvirul Alam et al. (2024)},
  year={2024},
  note={arXiv:2406.07599}
}

• arXiv: 2406.07599