synTSBench-eval

synTSBench-eval

SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series — Tan et al. (2025) (arXiv:2510.20273, 2025)

What this evaluates

Evaluates the ability of deep learning models to learn and forecast diverse temporal patterns (trends, periodicities, multivariate dependencies) in time series. It also probes model robustness against varying levels of Gaussian and non-Gaussian noise, as well as resilience to point and pulse anomalies.

Datasets

• SynTSBench — total ?; splits: test (-1); repo https://github.com/TanQitai/SynTSBench

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: $\frac{1}{n}\sum_{t=1}^{n}(y_{t}-\hat{y}_{t})^{2}$, where $y_t$ is the ground truth and $\hat{y}_t$ is the prediction. Lower is better.
• MAE — range: other
  • Mean Absolute Error: $\frac{1}{n}\sum_{t=1}^{n}|y_{t}-\hat{y}_{t}|$. Lower is better.
• MSETrue — range: other
  • Mean Squared Error computed against the underlying clean signal $x_t$ rather than the noisy observation $y_t$: $\frac{1}{n}\sum_{t=1}^{n}(x_{t}-\hat{y}_{t})^{2}$. Used to assess true denoising/forecasting capability under noise.

Input / output format

Input: A univariate or multivariate historical time series sequence (clean or corrupted with noise/anomalies) of variable length.

Output: A forecasted sequence of length H for the next time steps, where H ∈ {24, 48, 96, 192}.

Scoring recipe

def compute_metrics(pred, gold):
    mse = np.mean((pred - gold) ** 2)
    mae = np.mean(np.abs(pred - gold))
    return {'MSE': mse, 'MAE': mae}

def compute_mse_true(pred, clean_signal):
    return np.mean((pred - clean_signal) ** 2)

# Average results across all four forecasting horizons H ∈ {24, 48, 96, 192}

Common pitfalls

• Confusing MSEObs (error against noisy data) with MSETrue (error against clean signal) when evaluating robustness; MSEObs converges to ~1 at extreme noise and loses discriminative power.
• Failing to average results across all four forecasting horizons (H ∈ {24, 48, 96, 192}) as specified in the tables.
• Ignoring the theoretical optimum baseline (0 error) when comparing model performance, which reveals the fundamental gap in pattern learning.

Evidence (verbatim from paper)

For noise evaluation, we tested models across multiple SNR levels from clean data to extreme noise (SNR = -10dB). Beyond Gaussian noise, we also evaluated model robustness under diverse noise distributions including uniform noise, Laplace noise, t-distributions, and heavy-tailed Lévy stable distributions to assess model performance under various realistic noise characteristics (detailed results in [Table 12] in [Section B.5]). We measured both the error between predictions and noisy observations (MSEObs) and between predictions and the underlying clean signal (MSETrue):

Citation

@misc{tan2025synTSBench,
  title={SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2510.20273}
}

• arXiv: 2510.20273