Cesnet Timeseries24 Cl Eval

cesnet-timeseries24-cl-eval

Temporal Taskification in Streaming Continual Learning: A Source of Evaluation Instability — Filat et al. (2026) (arXiv:2604.21930, 2026)

What this evaluates

This evaluation probes the stability and performance of continual learning algorithms on multivariate time-series forecasting when the underlying data stream is partitioned into tasks with varying temporal granularities. It measures how sensitive forecasting accuracy, catastrophic forgetting, and backward transfer are to the choice of task boundaries and window lengths.

Datasets

• CESNET-Timeseries24 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Average MSE (primary) — range: other
  • Mean Squared Error between predicted and actual avg_duration values, averaged across all test samples and tasks. Reported values are multiplied by 10^3 for readability.
• Forgetting — range: other
  • Adapted from classification to regression: f_j = M_{T,j} - min_{k in {j,...,T-1}} M_{k,j}, where M_{i,j} is the MSE on task j after training on task i. Averaged over past tasks: (1/(T-1)) * sum(f_j). Higher values indicate more performance degradation on previous tasks.
• Backward Transfer (BWT) — range: other
  • Adapted to regression: (1/(T-1)) * sum(M_{j,j} - M_{T,j}), where M_{j,j} is MSE on task j after training on it, and M_{T,j} is MSE after training on all T tasks. Higher values indicate better retention of previous tasks.
• Boundary-Profile Stability (BPS) — range: other
  • Measures sensitivity of task boundaries using first-order Wasserstein distance between plasticity and stability profiles under ±1 day boundary perturbations. Coefficients for plasticity and stability are weighted equally (α=β=0.5). Lower values indicate more robust taskifications.

Input / output format

Input: Multivariate time series containing 12 features for 100 IP addresses, aggregated at 10-minute intervals. The model receives a 2-day context window of past observations.

Output: A single predicted value for the avg_duration feature at the next 10-minute timestep.

Scoring recipe

# M[i, j] = MSE on task j after training on task i
# T = total number of tasks
forgetting = 0.0
bwt = 0.0
for j in range(T - 1):
    f_j = M[T, j] - min(M[k, j] for k in range(j, T - 1))
    forgetting += f_j
    bwt += (M[j, j] - M[T, j])
forgetting /= (T - 1)
bwt /= (T - 1)
# Average MSE is computed directly on test predictions vs ground truth, then multiplied by 1000 for reporting.

Common pitfalls

• Forgetting and BWT are adapted from classification accuracy to regression MSE, which inverts the optimization direction (minimization vs maximization) and changes the interpretation of positive/negative values.
• Reported MSE values are scaled by 10^3 for readability; raw values are 1000x smaller.
• Task splits are strictly temporal and non-overlapping, with 80/10/10 train/val/test ratios applied per task window, not globally.

Evidence (verbatim from paper)

Performance is measured with average MSE (Mean Squared Error), Backward Transfer, and Forgetting. Since the latter two are traditionally defined for classification accuracy, we adapt them to the regression setting by defining $M_{i,j}$ as the MSE (Mean Squared Error) on task $j$ after training on task $i$. For a total of $T$ tasks, forgetting is computed as $f_{j}=M_{T,j}-\min_{k\in{j,\dots,T-1}}M_{k,j}$ and averaged over past tasks as $\mathrm{Forgetting}=\frac{1}{T-1}\sum_{j=1}^{T-1}f_{j}$. Similarly, Backward Transfer is defined as $\mathrm{BWT}=\frac{1}{T-1}\sum_{j=1}^{T-1}\left(M_{j,j}-M_{T,j}\right)$.

Citation

@misc{filat2026temporal,
  title={Temporal Taskification in Streaming Continual Learning: A Source of Evaluation Instability},
  author={Filat et al. (2026)},
  year={2026},
  note={arXiv:2604.21930}
}

• arXiv: 2604.21930