Ucr Augmented Eval

ucr-augmented-eval

Revisit Time Series Classification Benchmark: The Impact of Temporal Information for Classification — Zhang et al. (2025) (arXiv:2503.20264, 2025)

What this evaluates

Evaluates time series classifiers' reliance on temporal structure by measuring accuracy degradation when temporal alignment is disrupted via padding. It contrasts performance on the original UCR benchmark against a perturbed version to isolate the contribution of temporal correlations versus tabular features.

Datasets

• UCR Augmented — total ?; splits: train (-1), test (-1); repo https://github.com/YunruiZhang/Revisit-Time-Series-Classification-Benchmark

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total test set.
• p-value — range: [0, 1]
  • Wilcoxon signed-rank test comparing accuracy distributions on augmented vs. original datasets to assess significance of performance drop.
• mean ranking — range: [1, 7]
  • Average rank of each classifier across all datasets at a given augmentation level, based on test accuracy.

Input / output format

Input: Time series sequences (original and augmented with padding/misalignment of length l from 0.1n to 0.5n)

Output: Class label predictions for each time series instance

Scoring recipe

acc_orig = compute_accuracy(predictions_orig, gold_labels)
acc_aug = compute_accuracy(predictions_aug, gold_labels)
p_val = wilcoxon_signed_rank_test(acc_orig, acc_aug)
rankings = rank_classifiers_by_accuracy(acc_aug)
mean_rank = np.mean(rankings)

Common pitfalls

• Assuming high accuracy on the original UCR archive implies strong temporal modeling, as many datasets are effectively tabular.
• Ignoring phase-dependence in interval-based methods (e.g., CIF), which fail when distinguishing subsequences shift indices due to augmentation.
• Evaluating only on the original benchmark without perturbation, which masks reliance on non-temporal features.

Evidence (verbatim from paper)

To evaluate the level of the performance reduction for the classifiers, we performed the Wilcoxon signed-rank test [18] between the classifier's accuracy on the UCR augmented with different $l$ datasets and the accuracy on the original UCR archive datasets, with the alternative hypothesis being the reduction in classifiers' accuracy on the augmented UCR datasets compared to the original UCR archive.

Citation

@misc{zhang2025revisittimeseries,
  title={Revisit Time Series Classification Benchmark: The Impact of Temporal Information for Classification},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.20264}
}

• arXiv: 2503.20264