Romeo Vuln Detection Eval

romeo-vuln-detection-eval

ROMEO: Exploring Juliet through the Lens of Assembly Language — Brust et al. (2021) (arXiv:2112.06623, 2021)

What this evaluates

This benchmark evaluates binary vulnerability detection capabilities on assembly language representations of C/C++ functions. It probes whether models can identify security flaws (e.g., buffer overflows, integer overflows) by analyzing machine code semantics and call graph context.

Datasets

• ROMEO — total ?; splits: train (-1), val (-1), test (16764)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly classified instances (vulnerable vs. not vulnerable) out of the total test set.
• F1 score — range: percent
  • Harmonic mean of precision and recall for the binary vulnerability classification task.

Input / output format

Input: Tokenized assembly language instructions representing a single function, optionally augmented with call graph context from calling/called functions. Max 512 tokens per instance.

Output: Binary label: 'vulnerable' or 'not vulnerable'.

Scoring recipe

def compute_metrics(preds, gold):
    acc = sum(p == g for p, g in zip(preds, gold)) / len(gold) * 100
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'accuracy': acc, 'f1': f1 * 100}

Common pitfalls

• Comparing results across different evaluation granularities (e.g., program slices vs. whole functions) without normalization.
• Assuming random train/val/test splits; the dataset is split proportionally by CWE and flow variant to prevent data leakage, which limits cross-CWE generalization testing.
• Ignoring the context variant; performance drops significantly (~6.7% accuracy) when call graph context is removed, so both variants must be reported.

Evidence (verbatim from paper)

With context, the overall accuracy on the held-out test set is 96.9% and the overall F1 score is 94.0%. Without context, the accuracy and F1 score are 90.2% and 81.9%, respectively.

Citation

@misc{brust2021romeo,
  title={ROMEO: Exploring Juliet through the Lens of Assembly Language},
  author={Brust et al. (2021)},
  year={2021},
  note={arXiv:2112.06623}
}

• arXiv: 2112.06623