Concap Nids Eval

concap-nids-eval

ConCap: Practical Network Traffic Generation for (ML- and) Flow-based Intrusion Detection Systems — Verkerken et al. (2025) (arXiv:2509.16038, 2025)

What this evaluates

Evaluates the ability of machine learning and flow-based intrusion detection systems to accurately classify network traffic flows as benign or malicious. It probes the model's capacity to generalize across real-world benchmarks and synthetically generated, automatically labeled traffic for multi-step attack scenarios.

Datasets

• CICIDS17 — total ?; splits: train (-1), test (-1)
• ConCap ssh-patator — total ?; splits: train (-1), test (-1); repo https://github.com/idlab-discover/ConCap

Metrics

• tpr (primary) — range: [0, 1]
  • True Positive Rate (Recall) = TP / (TP + FN). Measures the proportion of actual malicious flows correctly identified by the detector.
• fpr — range: [0, 1]
  • False Positive Rate = FP / (FP + TN). Measures the proportion of benign flows incorrectly flagged as malicious.
• accuracy — range: [0, 1]
  • Accuracy = (TP + TN) / (TP + TN + FP + FN). Proportion of correctly classified flows out of the total test set.

Input / output format

Input: CSV file containing NetFlow records with features such as Src Port, Dst Port, Protocol, Flow Duration, and Number of packets.

Output: Binary classification label per row: 0 for Benign, 1 for Malicious.

Scoring recipe

tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
accuracy = (tp + tn) / len(gold) if len(gold) > 0 else 0.0
return {'tpr': tpr, 'fpr': fpr, 'accuracy': accuracy}

Common pitfalls

• AutoEncoders require explicit thresholding on reconstruction error to output binary classifications; using a threshold derived from a different dataset (e.g., xNIDS) can cause unexpectedly high FPR on ConCap data.
• LLM zero-shot prompting without context fails to detect malicious SSH bursts, as repeated connections alone do not inherently imply malicious intent without domain-specific heuristics or augmented examples.
• The evaluation uses an 80/20 split on both benign and malicious subsets independently; researchers must ensure the held-out 20% test set is strictly disjoint from training to avoid data leakage.

Evidence (verbatim from paper)

Specifically, when trained on ConCap’s generated data (which is always malicious), the AE-IDS achieves $tpr=0.999$ on the testing partition of ConCap’s generated data, and $tpr=0.987$ on the testing partition of malicious data of the same attack included in CICIDS17; conversely, when trained on the malicious data of CICIDS17, the AE-IDS achieves $tpr=0.955$ on the testing partition of CICIDS17, and $tpr=0.952$ on the testing partition of ConCap’s generated data. The $fpr$ is 0.478 on the former case, and 0.209 in the latter case: such an underwhelming result is because our AE-IDS uses the same thresholding mechanism used in xNIDS (note that AutoEncoders are not classifiers: to use them in a classification task, one must specify a threshold on the reconstruction error. We used the one of xNIDS, which was derived on a different dataset. To improve the $fpr$, one can simply change the threshold).

Citation

@misc{verkerken2025concap,
  title={ConCap: Practical Network Traffic Generation for (ML- and) Flow-based Intrusion Detection Systems},
  author={Verkerken et al. (2025)},
  year={2025},
  note={arXiv:2509.16038}
}

• arXiv: 2509.16038