Truck Driving Risk Eval

truck-driving-risk-eval

LIFT: Interpretable truck driving risk prediction with literature-informed fine-tuned LLMs — Xiao Hu et al. (2025) (arXiv:2510.22333, 2025)

What this evaluates

Binary classification of truck driving risk on specific highway segments based on historical behavior, short-term trip dynamics, and real-time traffic conditions. It probes a model's ability to predict forward collision warning events using a small, highly imbalanced dataset of real-world trajectory data.

Datasets

• Truck Driving Risk Dataset — total 1792; splits: train (896), test (896)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Calculated as (TP+TN)/(TP+TN+FP+FN), representing the proportion of correctly classified samples out of the total.
• Precision — range: [0, 1]
  • Calculated as TP/(TP+FP), measuring the proportion of predicted positive samples that are actually positive.
• Recall — range: [0, 1]
  • Calculated as TP/(TP+FN), measuring the proportion of actual positive samples correctly identified.
• F1-Score — range: [0, 1]
  • Calculated as 2*(Precision*Recall)/(Precision+Recall), providing the harmonic mean of precision and recall.

Input / output format

Input: 10 numerical features per trajectory: historical forward collision frequency, long-term speed std, historical trip ratio, trip forward collision frequency, trip lane departure frequency, trip average speed, trip speed std, traffic average speed, traffic speed std, traffic max speed.

Output: Binary risk label (0 for no forward collision warning, 1 for forward collision warning on target segments).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Dataset is highly imbalanced (74 positive vs 1717 negative samples); SMOTE oversampling during training may artificially inflate test performance if the test set is not stratified or if data leakage occurs.
• The 1:1 train/test split artificially balances the classes, masking the model's performance on real-world prevalence where negative samples vastly outnumber positives.
• Features are pre-selected based on elasticity analysis from prior work, limiting generalizability to other feature sets or traffic environments.

Evidence (verbatim from paper)

Before training, the dataset is divided into training set and testing set using 1:1 ratio. ... We used the overall accuracy metric to evaluate the prediction performance of the model on all samples. Considering that the prediction performance of the model on positive samples (real risk events) is more important than that of negative samples (no risk events) for the risk prediction problem, we also use precision, recall, and F1 score as evaluation metrics.

Citation

@misc{hu2025lift,
  title={LIFT: Interpretable truck driving risk prediction with literature-informed fine-tuned LLMs},
  author={Xiao Hu et al. (2025)},
  year={2025},
  note={arXiv:2510.22333}
}

• arXiv: 2510.22333