# Motime Eval

> motime-eval

- Skill: `qhjqhj00/motime-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/motime-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/motime-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/motime-eval

---


# motime-eval

> MoTime: A Dataset Suite for Multimodal Time Series Forecasting — Xin Zhou et al. (2025) (arXiv:2505.15072, 2025)

## What this evaluates

Evaluates multimodal time series forecasting models across two scenarios: varying-history forecasting (using long and short temporal sequences) and cold-start forecasting (predicting from minimal initial observations). It probes how well models leverage static external modalities like text and metadata to improve prediction accuracy, especially for sparse or short series.

## Datasets

- **PixelRec** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **AmazonReview** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **WikiPeople** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **Movielens** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **TaobaoFashion** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **Tianchi** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF
- **News** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/xinzzzhou/CCwTF

## Metrics

- `RMSE` **(primary)** — range: other
  - Root Mean Squared Error: sqrt(1/T * sum_{t=1}^T (y_t - y_hat_t)^2). Captures error in the original scale and is sensitive to large deviations.
- `WRMSPE` — range: other
  - Weighted Root Mean Squared Percentage Error: sqrt(1/T * sum_{t=1}^T (y_t - y_hat_t)^2) / (1/T * sum_{t=1}^T |y_t|). Normalizes RMSE by the mean absolute value of the ground truth for a scale-invariant view.

## Input / output format

**Input**: Historical time series values (7 steps for daily datasets, 6 steps for high-frequency News), optionally augmented with static external modalities (text, metadata, images) depending on the model variant.

**Output**: Forecasted time series values for a specified horizon (7 to 28 steps ahead for daily datasets, up to 12 steps ahead for News).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred):
    T = len(y_true)
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    wrmspe = np.sqrt(np.mean((y_true - y_pred) ** 2)) / np.mean(np.abs(y_true))
    return rmse, wrmspe
```

## Common pitfalls

- Do not normalize or scale the data before evaluation; scores must be computed on the original data scale to preserve error meaningfulness.
- Avoid using scaled metrics like RMSSE, as the test spans are long enough for absolute-scale metrics and RMSSE is difficult to interpret with varying forecast horizons.
- In cold-start forecasting, use only 7 previous daily steps or 6 previous 20-minute steps from relevant non-target entities, not the target entity itself.

## Evidence (verbatim from paper)

> We report two widely used metrics [18], $$ \mathrm {R M S E} = \sqrt {\frac {1}{T} \sum_ {t = 1} ^ {T} (y _ {t} - \hat {y} _ {t}) ^ {2}}, \quad \mathrm {W R M S P E} = \frac{\sqrt {\frac {1}{T} \sum t = 1 ^ {T} (y _ {t} - \hat {y} _ {t}) ^ {2}}}{\frac {1}{T} \sum t = 1 ^ {T} | y _ {t} |} $$ RMSE captures error in the original scale and is particularly sensitive to large deviations. WRMSPE normalizes RMSE by the mean absolute value of the ground truth, offering a scale-invariant view of forecasting quality.

## Citation

```bibtex
@misc{zhou2025motime,
  title={MoTime: A Dataset Suite for Multimodal Time Series Forecasting},
  author={Xin Zhou et al. (2025)},
  year={2025},
  note={arXiv:2505.15072}
}
```

- arXiv: 2505.15072