Hstf Trojan Detection Eval

hstf-trojan-detection-eval

HSTF-Model: an HTTP-based Trojan Detection Model via the Hierarchical Spatio-Temporal Features of Traffics — Jiang Xie et al. (2023) (arXiv:2309.03724, 2023)

What this evaluates

Evaluates a deep learning model's ability to detect HTTP-based Trojan malware in network traffic by analyzing hierarchical spatio-temporal features. It probes the model's binary classification accuracy, robustness to training set class imbalance, and cross-dataset generalization performance.

Datasets

• BTHT-2018 — total 4082588; splits: train (-1), test (-1)
• ISCX-2012 — total 245348; splits: train (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • F1 = 2 * (Precision * Recall) / (Precision + Recall), where Precision = TP/(TP+FP) and Recall = TP/(TP+FN).
• Precision — range: [0, 1]
  • Precision = TP / (TP + FP).
• Recall — range: [0, 1]
  • Recall = TP / (TP + FN).
• FPR — range: [0, 1]
  • FPR = FP / (TN + FP).
• TPR — range: [0, 1]
  • TPR = TP / (TP + FN) (equivalent to Recall).

Input / output format

Input: Preprocessed HTTP flow data represented as sequences of request/response packets, payload sequences, and statistical features (packet size, flow size).

Output: Binary classification label: 'Malicious' or 'Benign', derived from a probability score compared against a threshold λ.

Scoring recipe

def compute_metrics(tp, fp, tn, fn):
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    fpr = fp / (tn + fp) if (tn + fp) > 0 else 0.0
    tpr = recall
    return {'precision': precision, 'recall': recall, 'f1': f1, 'fpr': fpr, 'tpr': tpr}

Common pitfalls

• Test set is artificially balanced 1:1 (malicious:benign), which masks real-world extreme class imbalance and may inflate perceived practical utility.
• Generalization is evaluated by training on BTHT-2018 and testing on ISCX-2012, but the datasets differ in traffic types and collection environments, making direct performance comparison difficult.
• The decision threshold λ is mentioned but never explicitly defined or tuned, making exact reproduction of binary predictions challenging.

Evidence (verbatim from paper)

Precision and recall are used as primary evaluation indicators to verify the detection performance of the model, as shown in Eq(14). Fβ is also calculated as a comprehensive evaluation index. A represents the weight of P and R in this index. The larger β(>1) represents R is more important, and the smaller β(<1) represents P is more important. We set β=1 to show that both are equally important.

Citation

@misc{xie2023hstfmodel,
  title={HSTF-Model: an HTTP-based Trojan Detection Model via the Hierarchical Spatio-Temporal Features of Traffics},
  author={Jiang Xie et al. (2023)},
  year={2023},
  note={arXiv:2309.03724}
}

• arXiv: 2309.03724