Chattime Eval

chattime-eval

ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data — Chengsen Wang et al. (2024) (arXiv:2412.11376, 2024)

What this evaluates

Evaluates a multimodal time series foundation model on zero-shot forecasting, context-guided forecasting, and time series question answering. It probes the model's ability to handle discretized numerical time series alongside textual prompts for prediction and feature recognition without task-specific fine-tuning.

Datasets

• Electric, Exchange, Traffic, Weather, and ETT datasets — total ?; splits: train (-1), val (-1), test (-1)
• Context-guided multimodal datasets — total ?; splits: train (-1), val (-1), test (-1)
• Synthesized TSQA dataset — total ?; splits: test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values. Lower values indicate better performance.
• Accuracy — range: [0, 1]
  • Proportion of correctly identified time series features (trend, volatility, season, outlier). Higher values indicate better performance.
• Average Rank — range: other
  • Average of per-dataset/model ranks across all evaluated datasets or features. Lower values indicate better performance.

Input / output format

Input: Tokenized time series sequences (discretized with mark characters) optionally concatenated with textual context or questions. History length is set to 2–5× the prediction length.

Output: Forecasted time series values (as tokens) or natural language answers to time series feature questions.

Scoring recipe

def compute_mae(preds, gold):
    return np.mean(np.abs(preds - gold))

def compute_accuracy(preds, gold):
    return np.mean(preds == gold)

def compute_avg_rank(results_per_dataset):
    ranks = []
    for ds, scores in results_per_dataset.items():
        sorted_models = sorted(scores.items(), key=lambda x: x[1])
        ranks.append({m: r+1 for r, (m, _) in enumerate(sorted_models)})
    avg_ranks = {m: np.mean([r[m] for r in ranks]) for m in ranks[0]}
    return avg_ranks

Common pitfalls

• Datasets are chronologically split 6:2:2; random shuffling violates temporal integrity and causes data leakage.
• History length must be scaled to 2–5× prediction length to ensure ≥2 complete periods; fixed windows cause insufficient context or leakage.
• Closed-source baselines (e.g., TimeGPT) use official APIs, while open-source baselines use GitHub implementations with recommended hyperparameters, creating a slight evaluation asymmetry.

Evidence (verbatim from paper)

We report the Mean Absolute Error (MAE) as the evaluation metric, where lower values mean better performance. ... Given the nature of feature recognition, we report the accuracy (Acc) as evaluation metric, with higher scores indicating better performance.

Citation

@misc{wang2024chattime,
  title={ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data},
  author={Chengsen Wang et al. (2024)},
  year={2024},
  note={arXiv:2412.11376}
}

• arXiv: 2412.11376