Mobile Traffic Forecasting Eval

mobile-traffic-forecasting-eval

Multi-Service Mobile Traffic Forecasting via Convolutional Long Short-Term Memories — Chaoyun Zhang et al. (2019) (arXiv:1905.09771, 2019)

What this evaluates

Evaluates the ability of deep learning models to forecast multi-service mobile network traffic volumes at the antenna level over short time horizons (up to 1 hour). It probes spatiotemporal sequence modeling by requiring models to capture both spatial correlations across antennas and temporal dependencies in traffic patterns.

Datasets

• Real-world mobile traffic dataset (36 services, 800 antennas) — total ?; splits: test (-1)

Metrics

• MAE (primary) — range: other
  • Mean absolute error averaged over all services and antennas: MAE(t) = (1/|S||A|) * sum_{s,a} |predicted - ground_truth|.
• PSNR — range: other
  • Peak signal-to-noise ratio based on MSE: PSNR(t) = 20log10(d_max) - 10log10(MSE), where d_max is the highest traffic volume in the test set.
• SSIM — range: [0, 1]
  • Structural similarity index measuring perceived similarity between ground truth and predicted traffic snapshots, using L=2, k1=0.1, k2=0.3 for stabilization constants.
• NMAE — range: percent
  • Normalized MAE per service, dividing the absolute error by the total span of demand values generated by that service over the prediction horizon to enable cross-service comparability.

Input / output format

Input: 12 consecutive 5-minute mobile traffic snapshots (channels = services, spatial dimension = antennas)

Output: 12 consecutive 5-minute traffic volume predictions for the next hour

Scoring recipe

def evaluate(pred, gt):
    # pred, gt: (num_services, num_antennas, time_steps)
    mae = np.mean(np.abs(pred - gt))
    mse = np.mean((pred - gt) ** 2)
    d_max = np.max(gt)
    psnr = 20 * np.log10(d_max) - 10 * np.log10(mse)
    ssim = compute_ssim(gt, pred, L=2, k1=0.1, k2=0.3)
    nmae = []
    for s in range(pred.shape[0]):
        denom = np.sum(gt[s])
        nmae.append(np.mean(np.abs(pred[s] - gt[s]) / denom))
    return {'MAE': mae, 'PSNR': psnr, 'SSIM': ssim, 'NMAE': nmae}

Common pitfalls

• PSNR and SSIM are borrowed from image processing; applying them to traffic data requires correct handling of the dynamic range L and stabilization constants c1/c2.
• NMAE normalizes by the total span of demand values per service/antenna, not by the mean or maximum, which significantly affects cross-service comparability.
• Models are trained by minimizing MSE loss but evaluated with MAE/PSNR/SSIM, which can create an optimization-evaluation gap if not accounted for.

Evidence (verbatim from paper)

We evaluate the performance of our proposed S2S-ConvLSTM, along with that of all benchmarks, by means of the following three metrics. The mean absolute error (MAE) is usually used to measure the difference between two variables, but it is also employed as a measure of prediction accuracy: [formula]. The peak signal-to-noise ratio (PSNR) is often used to assess the quality of image reconstruction. Nonetheless, this metric has also been employed in networking scenarios, where mobile network traffic snapshots are treated similar to images. Structural similarity (SSIM) is traditionally employed for measuring the perceived similarity between uncompressed and compressed images or videos. We can take a similar approach to measure the similarity between ground truth traffic snapshots and their predicted counterparts

Citation

@misc{zhang2019multiservice,
  title={Multi-Service Mobile Traffic Forecasting via Convolutional Long Short-Term Memories},
  author={Chaoyun Zhang et al. (2019)},
  year={2019},
  note={arXiv:1905.09771}
}

• arXiv: 1905.09771