Caf 7m Eval

caf-7m-eval

Overcoming the Modality Gap in Context-Aided Forecasting — Zheng et al. (2026) (arXiv:2603.12451, 2026)

What this evaluates

Evaluates whether context-aided forecasting models can effectively leverage supplementary descriptive context to improve probabilistic time series forecasts, and tests generalization to out-of-domain real-world datasets.

Datasets

• CAF-7M — total ?; splits: HARD (-1), EASY (-1), ALL (-1); repo https://github.com/ServiceNow/DoubleCast
• CGTSF — total ?; splits: test (-1)
• GIFT-Eval — total ?; splits: test (-1)

Metrics

• CRPS (primary) — range: other
  • Continuous Ranked Probability Score, a proper scoring rule for probabilistic forecasts. Normalized by dividing the mean CRPS by the mean absolute value of the ground truth to prevent large values from dominating aggregation.
• Win Rate — range: [0, 1]
  • The ratio of evaluation windows where the model's CRPS is lower than Chronos' CRPS. Designed to be robust to score outliers.

Input / output format

Input: Historical time series values paired with descriptive textual context. Evaluated via zero-shot prompting or fine-tuned model inference.

Output: Probabilistic forecast distribution over the prediction horizon.

Scoring recipe

def compute_crps(pred_dist, actual):
    return proper_scoring_rule(pred_dist, actual)

def normalize_crps(crps_values, actuals):
    return np.mean(crps_values) / np.mean(np.abs(actuals))

def compute_win_rate(model_crps, baseline_crps):
    return np.mean(model_crps < baseline_crps)

Common pitfalls

• Using unverified synthetic contexts can falsely attribute poor performance to model inability rather than low context quality.
• Auto-regressive chunking on prediction lengths exceeding pretraining limits (e.g., >64 steps) causes distribution mismatch, skewing zero-shot results.
• Context swapping/omission ablations are required to verify actual context utilization, as models may ignore or misalign with provided text.

Evidence (verbatim from paper)

Since Direct Prompt produces a probabilistic forecast, we use the CRPS (gneiting2007strictly) as our primary metric, which is a proper scoring rule for probabilistic forecasts. To avoid large values dominating the aggregated results, we normalize the CRPS using the mean absolute values of the forecast ground truths. As a complementary metric that is robust to score outliers, we also compute the Win Rate: the ratio of windows where Direct Prompt has a lower CRPS than Chronos.

Citation

@misc{zheng2026overcoming,
  title={Overcoming the Modality Gap in Context-Aided Forecasting},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2603.12451}
}

• arXiv: 2603.12451