Motime Eval

motime-eval

MoTime: A Dataset Suite for Multimodal Time Series Forecasting — Xin Zhou et al. (2025) (arXiv:2505.15072, 2025)

What this evaluates

Evaluates multimodal time series forecasting models across two scenarios: varying-history forecasting (using long and short temporal sequences) and cold-start forecasting (predicting from minimal initial observations). It probes how well models leverage static external modalities like text and metadata to improve prediction accuracy, especially for sparse or short series.

Datasets

• PixelRec — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
• AmazonReview — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
• WikiPeople — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
• Movielens — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
• TaobaoFashion — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
• Tianchi — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
• News — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error: sqrt(1/T * sum_{t=1}^T (y_t - y_hat_t)^2). Captures error in the original scale and is sensitive to large deviations.
• WRMSPE — range: other
  • Weighted Root Mean Squared Percentage Error: sqrt(1/T * sum_{t=1}^T (y_t - y_hat_t)^2) / (1/T * sum_{t=1}^T |y_t|). Normalizes RMSE by the mean absolute value of the ground truth for a scale-invariant view.

Input / output format

Input: Historical time series values (7 steps for daily datasets, 6 steps for high-frequency News), optionally augmented with static external modalities (text, metadata, images) depending on the model variant.

Output: Forecasted time series values for a specified horizon (7 to 28 steps ahead for daily datasets, up to 12 steps ahead for News).

Scoring recipe

def compute_metrics(y_true, y_pred):
    T = len(y_true)
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    wrmspe = np.sqrt(np.mean((y_true - y_pred) ** 2)) / np.mean(np.abs(y_true))
    return rmse, wrmspe

Common pitfalls

• Do not normalize or scale the data before evaluation; scores must be computed on the original data scale to preserve error meaningfulness.
• Avoid using scaled metrics like RMSSE, as the test spans are long enough for absolute-scale metrics and RMSSE is difficult to interpret with varying forecast horizons.
• In cold-start forecasting, use only 7 previous daily steps or 6 previous 20-minute steps from relevant non-target entities, not the target entity itself.

Evidence (verbatim from paper)

We report two widely used metrics [18], $$ \mathrm {R M S E} = \sqrt {\frac {1}{T} \sum_ {t = 1} ^ {T} (y _ {t} - \hat {y} _ {t}) ^ {2}}, \quad \mathrm {W R M S P E} = \frac{\sqrt {\frac {1}{T} \sum t = 1 ^ {T} (y _ {t} - \hat {y} _ {t}) ^ {2}}}{\frac {1}{T} \sum t = 1 ^ {T} | y _ {t} |} $$ RMSE captures error in the original scale and is particularly sensitive to large deviations. WRMSPE normalizes RMSE by the mean absolute value of the ground truth, offering a scale-invariant view of forecasting quality.

Citation

@misc{zhou2025motime,
  title={MoTime: A Dataset Suite for Multimodal Time Series Forecasting},
  author={Xin Zhou et al. (2025)},
  year={2025},
  note={arXiv:2505.15072}
}

• arXiv: 2505.15072