Time Series Transfer Eval

time_series_transfer_eval

Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features? — Otović Erik et al. (arXiv:2201.04449, 2022)

What this evaluates

Evaluates the effectiveness of transfer learning on time series data by comparing pre-trained models against models trained from scratch across intra-domain and cross-domain settings. It probes whether shared temporal structures enable knowledge transfer and how dataset size and domain similarity affect predictive performance and training convergence.

Datasets

• LEN-DB — total ?; splits: train (-1), test (-1)
• SPEECH — total ?; splits: train (-1), test (-1)
• EMG — total ?; splits: train (-1), test (-1)
• S&P 500 — total ?; splits: train (-1), test (-1)
• LOMAX — total ?; splits: train (-1), test (-1)
• STEAD — total ?; splits: train (-1), test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values. Lower is better.
• weighted F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, weighted by the number of true instances for each class. Higher is better.
• convergence rate — range: other
  • Number of training epochs or iterations required for the model to reach a predefined performance threshold or stabilize. Lower is better.

Input / output format

Input: Time series sequences (optionally augmented with a 'stream max' input feature representing the maximum value in the sequence).

Output: Predicted continuous values for regression tasks, or class labels for classification tasks.

Scoring recipe

1. For each source-target-architecture triplet, train 70 models (10 LR multipliers × 7 reruns).
2. Select the best LR multiplier per run based on the primary metric.
3. Compute the average primary metric score across the 7 runs for both the TL model and the referent (scratch) model.
4. Calculate percentage difference: if MAE, diff = (ref - tl) / ref * 100; if F1, diff = (tl - ref) / ref * 100. Positive always indicates improvement.
5. Count wins (TL > ref) and losses across all triplets.
6. Apply a sign test to the win/loss counts for overall TL effectiveness.
7. For domain-pair analysis, compute the average percentage difference per pair, then apply a Wilcoxon signed-ranks test with Benjamini-Krieger-Yekutieli correction for multiple comparisons.

Common pitfalls

• Misinterpreting the sign convention for percentage difference: positive values always denote improvement (decrease in MAE or increase in F1), regardless of the metric type.
• Skipping the grid search and 7-run averaging steps before statistical testing, which violates the paper's protocol for isolating optimal hyperparameters and reducing variance.
• Assuming negative transfer is permanent; the paper explicitly notes that negative transfer effects diminish as the target dataset size increases.

Evidence (verbatim from paper)

Table 2: Performance gains when using stream max. Positive difference in the case of LEN-DB and S&P 500 represents a decrease in MAE, while positive difference in the case of SPEECH and EMG represents an increase in the weighted F1 score.

Citation

@misc{otovic2022intra,
  title={Intra-domain and cross-domain transfer learning for time series data -- How transferable are the features?},
  author={Otović Erik et al.},
  year={2022},
  note={arXiv:2201.04449}
}

• arXiv: 2201.04449