qhjqhj00 can-qa-eval
CAN-QA: A Question-Answering Benchmark for Reasoning over In-Vehicle CAN Traffic — Chen et al. (2026) (arXiv:2604.24935, 2026)
What this evaluates
Evaluates large language models' ability to perform structured reasoning over temporally segmented in-vehicle CAN traffic logs. It probes capabilities in temporal analysis, multi-condition inference, and behavioral interpretation for automotive cybersecurity forensics.
Datasets
- CAN-QA — total 33128; splits: test (-1); repo https://github.com/Kriiiiss/CAN-QA
Metrics
accuracy(primary) — range: percent- Calculated as the proportion of correctly answered questions out of the total number of questions. Evaluated separately for True/False (TF) and Multiple-Choice (MCQ) formats, as well as across ten reasoning categories.
Input / output format
Input: A temporally segmented window of in-vehicle CAN traffic logs paired with a natural-language question describing a specific traffic property or anomaly.
Output: For TF tasks: a binary 'True' or 'False' answer. For MCQ tasks: selection of the single most precise explanation from a set of multiple-choice options.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
return correct / len(gold) * 100
Common pitfalls
- Models often rely on coarse statistical cues (e.g., presence of many IDs) rather than verifying exact numerical thresholds, leading to errors in interval-based reasoning.
- MCQ tasks are significantly harder than TF tasks because models must discriminate between multiple plausible alternatives with subtle quantitative or structural differences, not just verify a single statement.
Evidence (verbatim from paper)
Figure 3 presents the zero-shot prediction accuracy of the selected LLMs on the TF and MCQ tasks. Across models, TF accuracy ranges from 47% to 59%, while MCQ accuracy ranges from 25% to 40%.
Citation
@misc{chen2026canqa,
title={CAN-QA: A Question-Answering Benchmark for Reasoning over In-Vehicle CAN Traffic},
author={Chen et al. (2026)},
year={2026},
note={arXiv:2604.24935}
}
- arXiv: 2604.24935