Csi Bert2 Eval

csi-bert2-eval

CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensing — Zijian Zhao et al. (2024) (arXiv:2412.06861, 2024)

What this evaluates

Evaluates a transformer-based framework for Channel State Information (CSI) time-series prediction and wireless sensing classification. It probes the model's ability to recover missing data, predict future CSI sequences, and classify human actions or environmental states from Wi-Fi signals.

Datasets

• WiGesture — total ?; splits: train (-1), test (-1); HF RS2002/WiGesture
• WiFall — total ?; splits: train (-1), test (-1); HF RS2002/WiFall
• WiCount — total ?; splits: train (-1), test (-1)
• CommPre — total 892; splits: train (-1), test (-1)

Metrics

• MSE — range: other
  • Mean Squared Error: average of squared differences between ground truth c and prediction c_hat across N packets and M subcarriers.
• SMAPE — range: percent
  • Symmetric Mean Absolute Percentage Error: average of 2*|c - c_hat| / (|c| + |c_hat|) across all elements.
• MAPE — range: percent
  • Mean Absolute Percentage Error: average of |c - c_hat| / (c + epsilon) across all elements.
• Accuracy (primary) — range: percent
  • Percentage of correctly classified samples out of the total test set for gesture recognition, fall detection, and people counting tasks.

Input / output format

Input: 1-second CSI time-series samples (100 time steps × 52 subcarriers). For prediction tasks, the input is the first 80 steps (0.8s) and the target is the subsequent 20 steps (0.2s). For classification, the full 100-step sequence is used.

Output: Predicted CSI values (complex magnitude/phase or real/imaginary components) for reconstruction/prediction, or discrete class labels for sensing tasks.

Scoring recipe

def compute_metrics(y_true, y_pred, task='prediction'):
    if task == 'prediction':
        mse = np.mean((y_true - y_pred) ** 2)
        smape = np.mean(2 * np.abs(y_true - y_pred) / (np.abs(y_true) + np.abs(y_pred) + 1e-8))
        mape = np.mean(np.abs(y_true - y_pred) / (np.abs(y_true) + 1e-8))
        return mse, smape, mape
    else:
        return np.mean(y_true == y_pred) * 100

Common pitfalls

• Recovery error must be calculated only on the 15% of deleted/masked packets, not the entire sequence.
• Distinguish between 'recover' (filling only masked positions) and 'replace' (overwriting the whole sequence) strategies when training downstream classifiers.
• The model is explicitly tested on discontinuous sequences and varying sampling rates, which breaks conventional fixed-rate baselines.

Evidence (verbatim from paper)

We employ mean squared error (MSE), symmetric mean absolute percentage error (SMAPE), and mean absolute percentage error (MAPE) to quantify the error: ... The metrics are calculated only on the 15% of the deleted CSI.

Citation

@misc{zhao2024csibert2,
  title={CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensing},
  author={Zijian Zhao et al. (2024)},
  year={2024},
  note={arXiv:2412.06861}
}

• arXiv: 2412.06861