Rfuav Eval

rfuav-eval

RFUAV: A Benchmark Dataset for Unmanned Aerial Vehicle Detection and Identification — Rui Shi et al. (2025) (arXiv:2503.09033, 2025)

What this evaluates

Evaluates deep learning models' ability to identify specific UAV models from radio-frequency signals by classifying time-frequency spectrograms. It probes robustness to varying signal-to-noise ratios (SNR) and sensitivity to preprocessing choices like color maps and frequency resolution.

Datasets

• RFUAV — total ?; splits: train (-1), val (-1); repo https://github.com/kitoweeknd/RFUAV

Metrics

• Acc (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted drone models out of the total number of spectrogram samples in the evaluation set.
• OA — range: percent
  • Overall accuracy, reported identically to Acc or as macro-averaged accuracy across classes. Stratified variants are computed for specific SNR ranges (e.g., OA_SNR≤-10).

Input / output format

Input: 2D spectrogram images generated from raw RF signals via Short-Time Fourier Transform (STFT) and mapped to colors using a specified colormap (CMAP).

Output: Discrete class label corresponding to one of the five UAV models (e.g., DJI AVATA2, DJI FPV COMBO, DJI MAVIC3 PRO, DJI MINI3, DJI MINI4 PRO).

Scoring recipe

def compute_acc(predictions, labels):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    return (correct / len(labels)) * 100

def compute_stratified_acc(predictions, labels, snr_mask):
    subset_preds = [p for p, m in zip(predictions, snr_mask) if m]
    subset_labels = [l for l, m in zip(labels, snr_mask) if m]
    return compute_acc(subset_preds, subset_labels)

Common pitfalls

• The evaluation set is labeled as 'validation' in the text but functions as the test set for reporting final accuracy.
• SNR levels are artificially adjusted using additive white Gaussian noise (AWGN), which may not capture the complex interference and fading of real-world RF environments.
• Results are highly sensitive to preprocessing parameters (CMAP and STFT points), meaning models trained on one preprocessing pipeline may not generalize to another.

Evidence (verbatim from paper)

The spectrograms of the raw signals and the SNR-adjusted signals at 20 dB serve as training data, whereas the spectrograms of the SNR adjusted signals with SNR levels ranging from -20 dB to 20 dB (in 2 dB increments) are used as validation data. The evaluation metrics include accuracy (Acc) and overall accuracy (OA). To facilitate a clearer understanding of Acc variations under different SNR levels, we visualize the experimental results in Fig.[6] and provide detailed numerical results in Table[1].

Citation

@misc{shi2025rfuav,
  title={RFUAV: A Benchmark Dataset for Unmanned Aerial Vehicle Detection and Identification},
  author={Rui Shi et al. (2025)},
  year={2025},
  note={arXiv:2503.09033}
}

• arXiv: 2503.09033